2024-01-21 02:53:14,134 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-01-21 02:53:14,438 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-01-21 02:53:14,551 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-01-21 02:53:14,620 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-21 02:53:14,628 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-21 02:53:14,641 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=340, ProcessCount=168, AvailableMemoryMB=13502 2024-01-21 02:53:14,643 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-21 02:53:14,643 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-01-21 02:53:14,783 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1013151981:84 , srv1065151499:84 , srv1067912997:84 , srv1072942886:84 , srv1117316827:84 , srv1158852362:84 , srv1175353302:84 , srv1182732270:84 , srv1202397417:84 , srv1206423623:84 , srv1251902114:84 , srv1260076443:84 , srv1295151531:84 , srv1342687050:84 , srv135054666:84 , srv1377326081:84 , srv145891984:84 , srv1478907062:84 , srv150131286:84 , srv150418659:84 , srv1508743825:84 , srv1532957398:84 , srv1562066898:84 , srv1579299330:84 , srv1593317893:84 , srv1610932194:84 , srv163286199:84 , srv1672293043:84 , srv1679492924:84 , srv1680465582:84 , srv1691505453:84 , srv1718938300:84 , srv1724350412:84 , srv1730799727:84 , srv1733880648:84 , srv1757422996:84 , srv1771836755:84 , srv1822744563:84 , srv1841733415:84 , srv1845864430:84 , srv1903684717:84 , srv1977418598:84 , srv1979075875:84 , srv198510505:84 , srv1987460341:84 , srv2031731326:84 , srv2046914462:84 , srv208827904:84 , srv2128370116:84 , srv213086949:84 , srv2138043708:84 , srv2140432248:84 , srv224432701:84 , srv236971729:84 , srv243249273:84 , srv246227694:84 , srv254385494:84 , srv266893066:84 , srv321337116:84 , srv334190697:84 , srv338698165:84 , srv341367275:84 , srv354054724:84 , srv380628855:84 , srv38999232:84 , srv400628360:84 , srv40277661:84 , srv407629705:84 , srv440242755:84 , srv457528471:84 , srv475489485:84 , srv494195767:84 , srv511590:84 , srv529954068:84 , srv54896974:84 , srv579540481:84 , srv598449377:84 , srv626845167:84 , srv65380103:84 , srv656416296:84 , srv658896340:84 , srv66956277:84 , srv681387762:84 , srv687370803:84 , srv691378868:84 , srv715641610:84 , srv724509459:84 , srv784709967:84 , srv806974588:84 , srv856756709:84 , srv89786528:84 , srv903560732:84 , srv951942719:84 , srv952905475:84 , srv976336190:84 , srv978053858:84 , srv983910889:84 , srv990036300:84 , srv992657147:84 , srv780798524:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-01-21 02:53:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-21 02:53:14,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv952905475=94, srv400628360=65, srv475489485=70, srv198510505=43, srv163286199=26, srv1117316827=4, srv334190697=59, srv440242755=68, srv1977418598=41, srv691378868=84, srv715641610=85, srv983910889=97, srv1295151531=12, srv2046914462=46, srv990036300=98, srv494195767=71, srv951942719=93, srv1532957398=21, srv1508743825=20, srv2031731326=45, srv598449377=76, srv254385494=56, srv579540481=75, srv66956277=81, srv1202397417=8, srv38999232=64, srv626845167=77, srv1182732270=7, srv40277661=66, srv65380103=78, srv135054666=14, srv145891984=16, srv658896340=80, srv1724350412=32, srv1691505453=30, srv780798524=87, srv1979075875=42, srv529954068=73, srv1718938300=31, srv1987460341=44, srv1251902114=10, srv1342687050=13, srv407629705=67, srv1260076443=11, srv89786528=91, srv208827904=47, srv1067912997=2, srv1733880648=34, srv1771836755=36, srv1593317893=24, srv2138043708=50, srv1680465582=29, srv354054724=62, srv1206423623=9, srv1757422996=35, srv1478907062=17, srv1065151499=1, srv1562066898=22, srv656416296=79, srv150418659=19, srv1377326081=15, srv784709967=88, srv1072942886=3, srv687370803=83, srv1845864430=39, srv266893066=57, srv243249273=54, srv150131286=18, srv992657147=99, srv54896974=74, srv1672293043=27, srv1610932194=25, srv806974588=89, srv213086949=49, srv681387762=82, srv976336190=95, srv1579299330=23, srv341367275=61, srv246227694=55, srv338698165=60, srv1841733415=38, srv2128370116=48, srv1822744563=37, srv511590=72, srv1903684717=40, srv724509459=86, srv224432701=52, srv1013151981=0, srv1158852362=5, srv236971729=53, srv321337116=58, srv1175353302=6, srv2140432248=51, srv856756709=90, srv457528471=69, srv978053858=96, srv1679492924=28, srv1730799727=33, srv380628855=63, srv903560732=92} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-21 02:53:14,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-21 02:53:14,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-21 02:53:14,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-21 02:53:14,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-21 02:53:14,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-21 02:53:14,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-21 02:53:14,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-21 02:53:14,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-21 02:53:14,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-21 02:53:14,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-21 02:53:14,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-21 02:53:14,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-21 02:53:14,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-21 02:53:14,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-21 02:53:14,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-21 02:53:14,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-21 02:53:14,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-21 02:53:14,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-21 02:53:14,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-21 02:53:14,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-21 02:53:14,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-21 02:53:14,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-21 02:53:14,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-21 02:53:14,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-21 02:53:14,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-21 02:53:14,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-21 02:53:14,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-21 02:53:14,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-21 02:53:14,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-21 02:53:14,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-21 02:53:14,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-21 02:53:14,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-21 02:53:14,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-21 02:53:14,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-21 02:53:14,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-21 02:53:14,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-21 02:53:14,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-21 02:53:14,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-21 02:53:14,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-21 02:53:14,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-21 02:53:14,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-21 02:53:14,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-21 02:53:14,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-21 02:53:14,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-21 02:53:14,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-21 02:53:14,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-21 02:53:14,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-21 02:53:14,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-21 02:53:14,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-21 02:53:14,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-21 02:53:14,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-21 02:53:14,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-21 02:53:14,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-21 02:53:14,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-21 02:53:14,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-21 02:53:14,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-21 02:53:14,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-21 02:53:14,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-21 02:53:14,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-21 02:53:14,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-21 02:53:14,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-21 02:53:14,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-21 02:53:14,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-21 02:53:14,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-21 02:53:14,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-21 02:53:14,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-21 02:53:14,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-21 02:53:14,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-21 02:53:14,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-21 02:53:14,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-21 02:53:14,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-21 02:53:14,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-21 02:53:14,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-21 02:53:14,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-21 02:53:14,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-21 02:53:14,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-21 02:53:14,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-21 02:53:14,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-21 02:53:14,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-21 02:53:14,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-21 02:53:14,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-21 02:53:14,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-21 02:53:14,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-21 02:53:14,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-21 02:53:14,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-21 02:53:14,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-21 02:53:14,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-21 02:53:14,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-21 02:53:14,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-21 02:53:14,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-21 02:53:14,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-21 02:53:14,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-21 02:53:14,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-21 02:53:14,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-21 02:53:14,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-21 02:53:14,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-21 02:53:14,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-21 02:53:14,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-21 02:53:14,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-21 02:53:14,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-21 02:53:14,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-21 02:53:15,351 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-21 02:53:15,351 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-21 02:53:15,352 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-21 02:53:15,772 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 833 ms to try 5000 different iterations. Found a solution that moves 4714 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17807461886031792. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04008222002509239, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053101347312660085, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5237777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04008222002509239, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17325, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24033333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-21 02:53:15,785 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2128370116:81 , srv1013151981:82 , srv1175353302:82 , srv1987460341:82 , srv681387762:82 , srv1158852362:83 , srv1182732270:83 , srv1610932194:83 , srv1822744563:83 , srv208827904:83 , srv213086949:83 , srv724509459:83 , srv784709967:83 , srv1260076443:84 , srv1730799727:84 , srv1977418598:84 , srv2046914462:84 , srv2138043708:84 , srv38999232:84 , srv626845167:84 , srv65380103:84 , srv691378868:84 , srv806974588:84 , srv1202397417:85 , srv1295151531:85 , srv1342687050:85 , srv150131286:85 , srv150418659:85 , srv1508743825:85 , srv1593317893:85 , srv1680465582:85 , srv1733880648:85 , srv1771836755:85 , srv236971729:85 , srv341367275:85 , srv354054724:85 , srv407629705:85 , srv475489485:85 , srv494195767:85 , srv529954068:85 , srv598449377:85 , srv656416296:85 , srv856756709:85 , srv145891984:86 , srv1532957398:86 , srv1757422996:86 , srv1903684717:86 , srv2031731326:86 , srv243249273:86 , srv266893066:86 , srv338698165:86 , srv380628855:86 , srv40277661:86 , srv457528471:86 , srv658896340:86 , srv66956277:86 , srv715641610:86 , srv951942719:86 , srv990036300:86 , srv1579299330:87 , srv1672293043:87 , srv1679492924:87 , srv1691505453:87 , srv1718938300:87 , srv224432701:87 , srv246227694:87 , srv254385494:87 , srv334190697:87 , srv54896974:87 , srv1251902114:88 , srv1562066898:88 , srv163286199:88 , srv1841733415:88 , srv1845864430:88 , srv400628360:88 , srv687370803:88 , srv89786528:88 , srv903560732:88 , srv952905475:88 , srv135054666:89 , srv1478907062:89 , srv1724350412:89 , srv1979075875:89 , srv2140432248:89 , srv511590:89 , srv976336190:89 , srv992657147:89 , srv1067912997:90 , srv1072942886:90 , srv1117316827:90 , srv1206423623:90 , srv198510505:90 , srv978053858:91 , srv983910889:91 , srv1065151499:92 , srv1377326081:93 , srv579540481:93 , srv440242755:95 , srv321337116:98 , srv780798524:446 } 2024-01-21 02:53:15,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-21 02:53:15,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv952905475=94, srv400628360=65, srv475489485=70, srv198510505=43, srv163286199=26, srv1117316827=4, srv334190697=59, srv440242755=68, srv1977418598=41, srv691378868=84, srv715641610=85, srv983910889=97, srv1295151531=12, srv2046914462=46, srv990036300=98, srv494195767=71, srv951942719=93, srv1532957398=21, srv1508743825=20, srv2031731326=45, srv598449377=76, srv254385494=56, srv579540481=75, srv66956277=81, srv1202397417=8, srv38999232=64, srv626845167=77, srv1182732270=7, srv40277661=66, srv65380103=78, srv135054666=14, srv145891984=16, srv658896340=80, srv1724350412=32, srv1691505453=30, srv780798524=87, srv1979075875=42, srv529954068=73, srv1718938300=31, srv1987460341=44, srv1251902114=10, srv1342687050=13, srv407629705=67, srv1260076443=11, srv89786528=91, srv208827904=47, srv1067912997=2, srv1733880648=34, srv1771836755=36, srv1593317893=24, srv2138043708=50, srv1680465582=29, srv354054724=62, srv1206423623=9, srv1757422996=35, srv1478907062=17, srv1065151499=1, srv1562066898=22, srv656416296=79, srv150418659=19, srv1377326081=15, srv784709967=88, srv1072942886=3, srv687370803=83, srv1845864430=39, srv266893066=57, srv243249273=54, srv150131286=18, srv992657147=99, srv54896974=74, srv1672293043=27, srv1610932194=25, srv806974588=89, srv213086949=49, srv681387762=82, srv976336190=95, srv1579299330=23, srv341367275=61, srv246227694=55, srv338698165=60, srv1841733415=38, srv2128370116=48, srv1822744563=37, srv511590=72, srv1903684717=40, srv724509459=86, srv224432701=52, srv1013151981=0, srv1158852362=5, srv236971729=53, srv321337116=58, srv1175353302=6, srv2140432248=51, srv856756709=90, srv457528471=69, srv978053858=96, srv1679492924=28, srv1730799727=33, srv380628855=63, srv903560732=92} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-21 02:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-21 02:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-21 02:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-21 02:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-21 02:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-21 02:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-21 02:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-21 02:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-21 02:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-21 02:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-21 02:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-21 02:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-21 02:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-21 02:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-21 02:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-21 02:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-21 02:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-21 02:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-21 02:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-21 02:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-21 02:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-21 02:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-21 02:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-21 02:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-21 02:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-21 02:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-21 02:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-21 02:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-21 02:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-21 02:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-21 02:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-21 02:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-21 02:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-21 02:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-21 02:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-21 02:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-21 02:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-21 02:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-21 02:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-21 02:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-21 02:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-21 02:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-21 02:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-21 02:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-21 02:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-21 02:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-21 02:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-21 02:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-21 02:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-21 02:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-21 02:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-21 02:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-21 02:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-21 02:53:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-21 02:53:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-21 02:53:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-21 02:53:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-21 02:53:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-21 02:53:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-21 02:53:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-21 02:53:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-21 02:53:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-21 02:53:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-21 02:53:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-21 02:53:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-21 02:53:15,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-21 02:53:15,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-21 02:53:15,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-21 02:53:15,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-21 02:53:15,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-21 02:53:15,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-21 02:53:15,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-21 02:53:15,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-21 02:53:15,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-21 02:53:15,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-21 02:53:15,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-21 02:53:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-21 02:53:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-21 02:53:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-21 02:53:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-21 02:53:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-21 02:53:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-21 02:53:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-21 02:53:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-21 02:53:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-21 02:53:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-21 02:53:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-21 02:53:15,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-21 02:53:15,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-21 02:53:15,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-21 02:53:15,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-21 02:53:15,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-21 02:53:15,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-21 02:53:15,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-21 02:53:15,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-21 02:53:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-21 02:53:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-21 02:53:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-21 02:53:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-21 02:53:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-21 02:53:15,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-21 02:53:15,935 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04008222002509239, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053101347312660085, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04008222002509239, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17325, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24033333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-21 02:53:15,935 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-21 02:53:15,936 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1780416107268209, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04008222002509239, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053101347312660085, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04008222002509239, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17325, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24033333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-21 02:53:16,173 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 331 ms to try 5000 different iterations. Found a solution that moves 2589 regions; Going from a computed imbalance of 0.1780416107268209 to a new imbalance of 0.014531295583310403. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020708247756095186); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032000280582383814, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2876666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020708247756095186); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04683333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-21 02:53:16,181 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv321337116:75 , srv1065151499:79 , srv440242755:79 , srv579540481:79 , srv1206423623:80 , srv1377326081:80 , srv198510505:80 , srv1072942886:81 , srv978053858:81 , srv983910889:81 , srv1117316827:82 , srv135054666:82 , srv1979075875:82 , srv1067912997:83 , srv1478907062:83 , srv1724350412:83 , srv246227694:83 , srv687370803:83 , srv952905475:83 , srv1251902114:84 , srv163286199:84 , srv1672293043:84 , srv1718938300:84 , srv1845864430:84 , srv266893066:84 , srv903560732:84 , srv951942719:84 , srv150418659:85 , srv1579299330:85 , srv1679492924:85 , srv1771836755:85 , srv2140432248:85 , srv254385494:85 , srv334190697:85 , srv338698165:85 , srv380628855:85 , srv658896340:85 , srv715641610:85 , srv89786528:85 , srv1295151531:86 , srv1342687050:86 , srv145891984:86 , srv1508743825:86 , srv1532957398:86 , srv1562066898:86 , srv1593317893:86 , srv1691505453:86 , srv1757422996:86 , srv2031731326:86 , srv236971729:86 , srv341367275:86 , srv354054724:86 , srv400628360:86 , srv40277661:86 , srv457528471:86 , srv475489485:86 , srv511590:86 , srv54896974:86 , srv656416296:86 , srv66956277:86 , srv856756709:86 , srv992657147:86 , srv150131286:87 , srv1733880648:87 , srv1841733415:87 , srv1903684717:87 , srv213086949:87 , srv224432701:87 , srv407629705:87 , srv598449377:87 , srv784709967:87 , srv976336190:87 , srv1202397417:88 , srv1260076443:88 , srv1680465582:88 , srv243249273:88 , srv38999232:88 , srv529954068:88 , srv65380103:88 , srv691378868:88 , srv724509459:88 , srv990036300:88 , srv1013151981:89 , srv1730799727:89 , srv1822744563:89 , srv2046914462:89 , srv208827904:89 , srv2128370116:89 , srv494195767:89 , srv1175353302:90 , srv1182732270:90 , srv1610932194:90 , srv626845167:90 , srv806974588:90 , srv1158852362:91 , srv1987460341:91 , srv2138043708:91 , srv681387762:91 , srv1977418598:92 , srv780798524:512 } 2024-01-21 02:53:16,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-21 02:53:16,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv952905475=94, srv400628360=65, srv475489485=70, srv198510505=43, srv163286199=26, srv1117316827=4, srv334190697=59, srv440242755=68, srv1977418598=41, srv691378868=84, srv715641610=85, srv983910889=97, srv1295151531=12, srv2046914462=46, srv990036300=98, srv494195767=71, srv951942719=93, srv1532957398=21, srv1508743825=20, srv2031731326=45, srv598449377=76, srv254385494=56, srv579540481=75, srv66956277=81, srv1202397417=8, srv38999232=64, srv626845167=77, srv1182732270=7, srv40277661=66, srv65380103=78, srv135054666=14, srv145891984=16, srv658896340=80, srv1724350412=32, srv1691505453=30, srv780798524=87, srv1979075875=42, srv529954068=73, srv1718938300=31, srv1987460341=44, srv1251902114=10, srv1342687050=13, srv407629705=67, srv1260076443=11, srv89786528=91, srv208827904=47, srv1067912997=2, srv1733880648=34, srv1771836755=36, srv1593317893=24, srv2138043708=50, srv1680465582=29, srv354054724=62, srv1206423623=9, srv1757422996=35, srv1478907062=17, srv1065151499=1, srv1562066898=22, srv656416296=79, srv150418659=19, srv1377326081=15, srv784709967=88, srv1072942886=3, srv687370803=83, srv1845864430=39, srv266893066=57, srv243249273=54, srv150131286=18, srv992657147=99, srv54896974=74, srv1672293043=27, srv1610932194=25, srv806974588=89, srv213086949=49, srv681387762=82, srv976336190=95, srv1579299330=23, srv341367275=61, srv246227694=55, srv338698165=60, srv1841733415=38, srv2128370116=48, srv1822744563=37, srv511590=72, srv1903684717=40, srv724509459=86, srv224432701=52, srv1013151981=0, srv1158852362=5, srv236971729=53, srv321337116=58, srv1175353302=6, srv2140432248=51, srv856756709=90, srv457528471=69, srv978053858=96, srv1679492924=28, srv1730799727=33, srv380628855=63, srv903560732=92} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-21 02:53:16,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-21 02:53:16,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-21 02:53:16,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-21 02:53:16,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-21 02:53:16,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-21 02:53:16,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-21 02:53:16,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-21 02:53:16,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-21 02:53:16,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-21 02:53:16,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-21 02:53:16,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-21 02:53:16,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-21 02:53:16,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-21 02:53:16,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-21 02:53:16,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-21 02:53:16,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-21 02:53:16,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-21 02:53:16,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-21 02:53:16,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-21 02:53:16,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-21 02:53:16,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-21 02:53:16,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-21 02:53:16,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-21 02:53:16,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-21 02:53:16,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-21 02:53:16,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-21 02:53:16,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-21 02:53:16,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-21 02:53:16,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-21 02:53:16,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-21 02:53:16,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-21 02:53:16,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-21 02:53:16,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-21 02:53:16,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-21 02:53:16,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-21 02:53:16,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-21 02:53:16,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-21 02:53:16,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-21 02:53:16,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-21 02:53:16,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-21 02:53:16,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-21 02:53:16,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-21 02:53:16,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-21 02:53:16,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-21 02:53:16,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-21 02:53:16,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-21 02:53:16,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-21 02:53:16,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-21 02:53:16,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-21 02:53:16,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-21 02:53:16,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-21 02:53:16,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-21 02:53:16,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-21 02:53:16,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-21 02:53:16,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-21 02:53:16,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-21 02:53:16,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-21 02:53:16,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-21 02:53:16,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-21 02:53:16,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-21 02:53:16,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-21 02:53:16,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-21 02:53:16,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-21 02:53:16,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-21 02:53:16,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-21 02:53:16,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-21 02:53:16,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-21 02:53:16,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-21 02:53:16,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-21 02:53:16,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-21 02:53:16,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-21 02:53:16,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-21 02:53:16,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-21 02:53:16,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-21 02:53:16,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-21 02:53:16,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-21 02:53:16,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-21 02:53:16,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-21 02:53:16,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-21 02:53:16,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-21 02:53:16,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-21 02:53:16,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-21 02:53:16,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-21 02:53:16,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-21 02:53:16,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-21 02:53:16,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-21 02:53:16,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-21 02:53:16,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-21 02:53:16,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-21 02:53:16,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-21 02:53:16,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-21 02:53:16,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-21 02:53:16,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-21 02:53:16,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-21 02:53:16,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-21 02:53:16,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-21 02:53:16,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-21 02:53:16,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-21 02:53:16,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-21 02:53:16,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-21 02:53:16,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-21 02:53:16,318 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020708247756095186); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032000280582383814, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020708247756095186); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04683333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-21 02:53:16,318 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-21 02:53:16,319 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014513167017840802, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020708247756095186); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032000280582383814, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020708247756095186); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04683333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-21 02:53:16,539 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 314 ms to try 5000 different iterations. Found a solution that moves 1043 regions; Going from a computed imbalance of 0.014513167017840802 to a new imbalance of 8.94742072027276E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0052019929608213104); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006581953356624206); MoveCostFunction : (multiplier=7.0, imbalance=0.11588888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0052019929608213104); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-21 02:53:16,542 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv976336190:81 , srv1977418598:82 , srv511590:82 , srv1158852362:83 , srv1841733415:83 , srv2138043708:83 , srv2140432248:83 , srv224432701:83 , srv400628360:83 , srv494195767:83 , srv992657147:83 , srv1067912997:84 , srv1182732270:84 , srv1377326081:84 , srv1562066898:84 , srv1680465582:84 , srv1730799727:84 , srv1987460341:84 , srv243249273:84 , srv440242755:84 , srv626845167:84 , srv806974588:84 , srv89786528:84 , srv1175353302:85 , srv1202397417:85 , srv1260076443:85 , srv1478907062:85 , srv1532957398:85 , srv1579299330:85 , srv1610932194:85 , srv1691505453:85 , srv1724350412:85 , srv1733880648:85 , srv1757422996:85 , srv1822744563:85 , srv1903684717:85 , srv2046914462:85 , srv208827904:85 , srv321337116:85 , srv334190697:85 , srv38999232:85 , srv40277661:85 , srv407629705:85 , srv529954068:85 , srv54896974:85 , srv681387762:85 , srv691378868:85 , srv903560732:85 , srv990036300:85 , srv1065151499:86 , srv1117316827:86 , srv1251902114:86 , srv135054666:86 , srv145891984:86 , srv150131286:86 , srv1508743825:86 , srv163286199:86 , srv1672293043:86 , srv1679492924:86 , srv1718938300:86 , srv1845864430:86 , srv236971729:86 , srv254385494:86 , srv341367275:86 , srv354054724:86 , srv457528471:86 , srv475489485:86 , srv579540481:86 , srv598449377:86 , srv65380103:86 , srv656416296:86 , srv66956277:86 , srv687370803:86 , srv715641610:86 , srv856756709:86 , srv952905475:86 , srv978053858:86 , srv1013151981:87 , srv1072942886:87 , srv1206423623:87 , srv1295151531:87 , srv1342687050:87 , srv1771836755:87 , srv1979075875:87 , srv198510505:87 , srv2031731326:87 , srv2128370116:87 , srv213086949:87 , srv246227694:87 , srv338698165:87 , srv380628855:87 , srv658896340:87 , srv724509459:87 , srv983910889:87 , srv150418659:88 , srv1593317893:88 , srv266893066:88 , srv784709967:88 , srv951942719:88 , srv780798524:546 } 2024-01-21 02:53:16,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-21 02:53:16,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv952905475=94, srv400628360=65, srv475489485=70, srv198510505=43, srv163286199=26, srv1117316827=4, srv334190697=59, srv440242755=68, srv1977418598=41, srv691378868=84, srv715641610=85, srv983910889=97, srv1295151531=12, srv2046914462=46, srv990036300=98, srv494195767=71, srv951942719=93, srv1532957398=21, srv1508743825=20, srv2031731326=45, srv598449377=76, srv254385494=56, srv579540481=75, srv66956277=81, srv1202397417=8, srv38999232=64, srv626845167=77, srv1182732270=7, srv40277661=66, srv65380103=78, srv135054666=14, srv145891984=16, srv658896340=80, srv1724350412=32, srv1691505453=30, srv780798524=87, srv1979075875=42, srv529954068=73, srv1718938300=31, srv1987460341=44, srv1251902114=10, srv1342687050=13, srv407629705=67, srv1260076443=11, srv89786528=91, srv208827904=47, srv1067912997=2, srv1733880648=34, srv1771836755=36, srv1593317893=24, srv2138043708=50, srv1680465582=29, srv354054724=62, srv1206423623=9, srv1757422996=35, srv1478907062=17, srv1065151499=1, srv1562066898=22, srv656416296=79, srv150418659=19, srv1377326081=15, srv784709967=88, srv1072942886=3, srv687370803=83, srv1845864430=39, srv266893066=57, srv243249273=54, srv150131286=18, srv992657147=99, srv54896974=74, srv1672293043=27, srv1610932194=25, srv806974588=89, srv213086949=49, srv681387762=82, srv976336190=95, srv1579299330=23, srv341367275=61, srv246227694=55, srv338698165=60, srv1841733415=38, srv2128370116=48, srv1822744563=37, srv511590=72, srv1903684717=40, srv724509459=86, srv224432701=52, srv1013151981=0, srv1158852362=5, srv236971729=53, srv321337116=58, srv1175353302=6, srv2140432248=51, srv856756709=90, srv457528471=69, srv978053858=96, srv1679492924=28, srv1730799727=33, srv380628855=63, srv903560732=92} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-21 02:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-21 02:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-21 02:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-21 02:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-21 02:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-21 02:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-21 02:53:16,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-21 02:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-21 02:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-21 02:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-21 02:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-21 02:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-21 02:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-21 02:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-21 02:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-21 02:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-21 02:53:16,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-21 02:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-21 02:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-21 02:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-21 02:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-21 02:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-21 02:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-21 02:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-21 02:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-21 02:53:16,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-21 02:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-21 02:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-21 02:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-21 02:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-21 02:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-21 02:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-21 02:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-21 02:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-21 02:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-21 02:53:16,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-21 02:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-21 02:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-21 02:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-21 02:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-21 02:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-21 02:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-21 02:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-21 02:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-21 02:53:16,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-21 02:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-21 02:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-21 02:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-21 02:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-21 02:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-21 02:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-21 02:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-21 02:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-21 02:53:16,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-21 02:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-21 02:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-21 02:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-21 02:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-21 02:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-21 02:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-21 02:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-21 02:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-21 02:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-21 02:53:16,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-21 02:53:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-21 02:53:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-21 02:53:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-21 02:53:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-21 02:53:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-21 02:53:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-21 02:53:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-21 02:53:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-21 02:53:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-21 02:53:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-21 02:53:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-21 02:53:16,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-21 02:53:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-21 02:53:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-21 02:53:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-21 02:53:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-21 02:53:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-21 02:53:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-21 02:53:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-21 02:53:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-21 02:53:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-21 02:53:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-21 02:53:16,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-21 02:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-21 02:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-21 02:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-21 02:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-21 02:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-21 02:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-21 02:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-21 02:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-21 02:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-21 02:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-21 02:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-21 02:53:16,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-21 02:53:16,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-21 02:53:16,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-21 02:53:16,689 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0052019929608213104); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006581953356624206); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0052019929608213104); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-21 02:53:16,690 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-21 02:53:16,690 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.874388299319527E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0052019929608213104); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006581953356624206); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0052019929608213104); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-21 02:53:17,020 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 433 ms to try 5000 different iterations. Found a solution that moves 253 regions; Going from a computed imbalance of 8.874388299319527E-4 to a new imbalance of 2.8229653016928E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.02811111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-21 02:53:17,021 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1072942886:83 , srv2031731326:83 , srv380628855:83 , srv457528471:83 , srv598449377:83 , srv983910889:83 , srv1013151981:84 , srv1117316827:84 , srv1202397417:84 , srv1206423623:84 , srv1251902114:84 , srv1260076443:84 , srv1295151531:84 , srv1342687050:84 , srv145891984:84 , srv150131286:84 , srv150418659:84 , srv1532957398:84 , srv1562066898:84 , srv1593317893:84 , srv163286199:84 , srv1672293043:84 , srv1679492924:84 , srv1691505453:84 , srv1718938300:84 , srv1757422996:84 , srv1841733415:84 , srv1845864430:84 , srv1903684717:84 , srv1977418598:84 , srv1979075875:84 , srv2138043708:84 , srv254385494:84 , srv266893066:84 , srv321337116:84 , srv338698165:84 , srv440242755:84 , srv529954068:84 , srv54896974:84 , srv579540481:84 , srv626845167:84 , srv658896340:84 , srv66956277:84 , srv681387762:84 , srv724509459:84 , srv784709967:84 , srv806974588:84 , srv951942719:84 , srv978053858:84 , srv990036300:84 , srv1067912997:85 , srv1158852362:85 , srv1175353302:85 , srv1182732270:85 , srv135054666:85 , srv1377326081:85 , srv1478907062:85 , srv1508743825:85 , srv1579299330:85 , srv1610932194:85 , srv1680465582:85 , srv1724350412:85 , srv1730799727:85 , srv1733880648:85 , srv1771836755:85 , srv1822744563:85 , srv198510505:85 , srv2046914462:85 , srv208827904:85 , srv2128370116:85 , srv213086949:85 , srv2140432248:85 , srv224432701:85 , srv236971729:85 , srv243249273:85 , srv246227694:85 , srv334190697:85 , srv341367275:85 , srv354054724:85 , srv38999232:85 , srv400628360:85 , srv40277661:85 , srv407629705:85 , srv475489485:85 , srv494195767:85 , srv511590:85 , srv65380103:85 , srv687370803:85 , srv691378868:85 , srv715641610:85 , srv856756709:85 , srv89786528:85 , srv903560732:85 , srv952905475:85 , srv976336190:85 , srv992657147:85 , srv1065151499:86 , srv1987460341:86 , srv656416296:86 , srv780798524:638 } 2024-01-21 02:53:17,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-21 02:53:17,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv952905475=94, srv400628360=65, srv475489485=70, srv198510505=43, srv163286199=26, srv1117316827=4, srv334190697=59, srv440242755=68, srv1977418598=41, srv691378868=84, srv715641610=85, srv983910889=97, srv1295151531=12, srv2046914462=46, srv990036300=98, srv494195767=71, srv951942719=93, srv1532957398=21, srv1508743825=20, srv2031731326=45, srv598449377=76, srv254385494=56, srv579540481=75, srv66956277=81, srv1202397417=8, srv38999232=64, srv626845167=77, srv1182732270=7, srv40277661=66, srv65380103=78, srv135054666=14, srv145891984=16, srv658896340=80, srv1724350412=32, srv1691505453=30, srv780798524=87, srv1979075875=42, srv529954068=73, srv1718938300=31, srv1987460341=44, srv1251902114=10, srv1342687050=13, srv407629705=67, srv1260076443=11, srv89786528=91, srv208827904=47, srv1067912997=2, srv1733880648=34, srv1771836755=36, srv1593317893=24, srv2138043708=50, srv1680465582=29, srv354054724=62, srv1206423623=9, srv1757422996=35, srv1478907062=17, srv1065151499=1, srv1562066898=22, srv656416296=79, srv150418659=19, srv1377326081=15, srv784709967=88, srv1072942886=3, srv687370803=83, srv1845864430=39, srv266893066=57, srv243249273=54, srv150131286=18, srv992657147=99, srv54896974=74, srv1672293043=27, srv1610932194=25, srv806974588=89, srv213086949=49, srv681387762=82, srv976336190=95, srv1579299330=23, srv341367275=61, srv246227694=55, srv338698165=60, srv1841733415=38, srv2128370116=48, srv1822744563=37, srv511590=72, srv1903684717=40, srv724509459=86, srv224432701=52, srv1013151981=0, srv1158852362=5, srv236971729=53, srv321337116=58, srv1175353302=6, srv2140432248=51, srv856756709=90, srv457528471=69, srv978053858=96, srv1679492924=28, srv1730799727=33, srv380628855=63, srv903560732=92} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-21 02:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-21 02:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-21 02:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-21 02:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-21 02:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-21 02:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-21 02:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-21 02:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-21 02:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-21 02:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-21 02:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-21 02:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-21 02:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-21 02:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-21 02:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-21 02:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-21 02:53:17,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-21 02:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-21 02:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-21 02:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-21 02:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-21 02:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-21 02:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-21 02:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-21 02:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-21 02:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-21 02:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-21 02:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-21 02:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-21 02:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-21 02:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-21 02:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-21 02:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-21 02:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-21 02:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-21 02:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-21 02:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-21 02:53:17,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-21 02:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-21 02:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-21 02:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-21 02:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-21 02:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-21 02:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-21 02:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-21 02:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-21 02:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-21 02:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-21 02:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-21 02:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-21 02:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-21 02:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-21 02:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-21 02:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-21 02:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-21 02:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-21 02:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-21 02:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-21 02:53:17,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-21 02:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-21 02:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-21 02:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-21 02:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-21 02:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-21 02:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-21 02:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-21 02:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-21 02:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-21 02:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-21 02:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-21 02:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-21 02:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-21 02:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-21 02:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-21 02:53:17,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-21 02:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-21 02:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-21 02:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-21 02:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-21 02:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-21 02:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-21 02:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-21 02:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-21 02:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-21 02:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-21 02:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-21 02:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-21 02:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-21 02:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-21 02:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-21 02:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-21 02:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-21 02:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-21 02:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-21 02:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-21 02:53:17,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-21 02:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-21 02:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-21 02:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-21 02:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-21 02:53:17,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-21 02:53:17,142 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-21 02:53:17,142 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-01-21 02:53:17,142 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.6458109152961765E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-01-21 02:53:17,356 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 306 ms to try 5000 different iterations. Found a solution that moves 28 regions; Going from a computed imbalance of 2.6458109152961765E-5 to a new imbalance of 7.269299486807925E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.003111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-21 02:53:17,357 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1065151499:83 , srv1593317893:83 , srv1610932194:83 , srv1987460341:83 , srv2046914462:83 , srv243249273:83 , srv65380103:83 , srv656416296:83 , srv990036300:83 , srv992657147:83 , srv1013151981:84 , srv1067912997:84 , srv1117316827:84 , srv1158852362:84 , srv1175353302:84 , srv1182732270:84 , srv1202397417:84 , srv1251902114:84 , srv1295151531:84 , srv1342687050:84 , srv135054666:84 , srv1377326081:84 , srv145891984:84 , srv1478907062:84 , srv150131286:84 , srv150418659:84 , srv1508743825:84 , srv1562066898:84 , srv1579299330:84 , srv163286199:84 , srv1679492924:84 , srv1680465582:84 , srv1691505453:84 , srv1724350412:84 , srv1730799727:84 , srv1733880648:84 , srv1771836755:84 , srv1822744563:84 , srv1841733415:84 , srv1845864430:84 , srv1903684717:84 , srv1977418598:84 , srv1979075875:84 , srv198510505:84 , srv2031731326:84 , srv208827904:84 , srv2128370116:84 , srv213086949:84 , srv2138043708:84 , srv2140432248:84 , srv224432701:84 , srv236971729:84 , srv246227694:84 , srv254385494:84 , srv266893066:84 , srv321337116:84 , srv334190697:84 , srv338698165:84 , srv341367275:84 , srv354054724:84 , srv38999232:84 , srv400628360:84 , srv40277661:84 , srv407629705:84 , srv440242755:84 , srv475489485:84 , srv494195767:84 , srv511590:84 , srv529954068:84 , srv54896974:84 , srv579540481:84 , srv626845167:84 , srv658896340:84 , srv66956277:84 , srv681387762:84 , srv687370803:84 , srv691378868:84 , srv715641610:84 , srv724509459:84 , srv784709967:84 , srv806974588:84 , srv856756709:84 , srv89786528:84 , srv903560732:84 , srv951942719:84 , srv952905475:84 , srv976336190:84 , srv978053858:84 , srv983910889:84 , srv1072942886:85 , srv1206423623:85 , srv1260076443:85 , srv1532957398:85 , srv1672293043:85 , srv1718938300:85 , srv1757422996:85 , srv380628855:85 , srv457528471:85 , srv598449377:85 , srv780798524:684 } 2024-01-21 02:53:17,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-21 02:53:17,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv952905475=94, srv400628360=65, srv475489485=70, srv198510505=43, srv163286199=26, srv1117316827=4, srv334190697=59, srv440242755=68, srv1977418598=41, srv691378868=84, srv715641610=85, srv983910889=97, srv1295151531=12, srv2046914462=46, srv990036300=98, srv494195767=71, srv951942719=93, srv1532957398=21, srv1508743825=20, srv2031731326=45, srv598449377=76, srv254385494=56, srv579540481=75, srv66956277=81, srv1202397417=8, srv38999232=64, srv626845167=77, srv1182732270=7, srv40277661=66, srv65380103=78, srv135054666=14, srv145891984=16, srv658896340=80, srv1724350412=32, srv1691505453=30, srv780798524=87, srv1979075875=42, srv529954068=73, srv1718938300=31, srv1987460341=44, srv1251902114=10, srv1342687050=13, srv407629705=67, srv1260076443=11, srv89786528=91, srv208827904=47, srv1067912997=2, srv1733880648=34, srv1771836755=36, srv1593317893=24, srv2138043708=50, srv1680465582=29, srv354054724=62, srv1206423623=9, srv1757422996=35, srv1478907062=17, srv1065151499=1, srv1562066898=22, srv656416296=79, srv150418659=19, srv1377326081=15, srv784709967=88, srv1072942886=3, srv687370803=83, srv1845864430=39, srv266893066=57, srv243249273=54, srv150131286=18, srv992657147=99, srv54896974=74, srv1672293043=27, srv1610932194=25, srv806974588=89, srv213086949=49, srv681387762=82, srv976336190=95, srv1579299330=23, srv341367275=61, srv246227694=55, srv338698165=60, srv1841733415=38, srv2128370116=48, srv1822744563=37, srv511590=72, srv1903684717=40, srv724509459=86, srv224432701=52, srv1013151981=0, srv1158852362=5, srv236971729=53, srv321337116=58, srv1175353302=6, srv2140432248=51, srv856756709=90, srv457528471=69, srv978053858=96, srv1679492924=28, srv1730799727=33, srv380628855=63, srv903560732=92} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-21 02:53:17,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-21 02:53:17,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-21 02:53:17,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-21 02:53:17,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-21 02:53:17,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-21 02:53:17,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-21 02:53:17,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-21 02:53:17,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-01-21 02:53:17,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-01-21 02:53:17,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-01-21 02:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-01-21 02:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-01-21 02:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-01-21 02:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-01-21 02:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-01-21 02:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-01-21 02:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-01-21 02:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-01-21 02:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-01-21 02:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-01-21 02:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-01-21 02:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-01-21 02:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-01-21 02:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-01-21 02:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-01-21 02:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-01-21 02:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-01-21 02:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-01-21 02:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-01-21 02:53:17,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-01-21 02:53:17,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-01-21 02:53:17,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-01-21 02:53:17,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-01-21 02:53:17,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-01-21 02:53:17,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-01-21 02:53:17,484 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-21 02:53:17,484 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1065151499:83 , srv1593317893:83 , srv1610932194:83 , srv1987460341:83 , srv2046914462:83 , srv243249273:83 , srv65380103:83 , srv656416296:83 , srv990036300:83 , srv992657147:83 , srv1013151981:84 , srv1067912997:84 , srv1117316827:84 , srv1158852362:84 , srv1175353302:84 , srv1182732270:84 , srv1202397417:84 , srv1251902114:84 , srv1295151531:84 , srv1342687050:84 , srv135054666:84 , srv1377326081:84 , srv145891984:84 , srv1478907062:84 , srv150131286:84 , srv150418659:84 , srv1508743825:84 , srv1562066898:84 , srv1579299330:84 , srv163286199:84 , srv1679492924:84 , srv1680465582:84 , srv1691505453:84 , srv1724350412:84 , srv1730799727:84 , srv1733880648:84 , srv1771836755:84 , srv1822744563:84 , srv1841733415:84 , srv1845864430:84 , srv1903684717:84 , srv1977418598:84 , srv1979075875:84 , srv198510505:84 , srv2031731326:84 , srv208827904:84 , srv2128370116:84 , srv213086949:84 , srv2138043708:84 , srv2140432248:84 , srv224432701:84 , srv236971729:84 , srv246227694:84 , srv254385494:84 , srv266893066:84 , srv321337116:84 , srv334190697:84 , srv338698165:84 , srv341367275:84 , srv354054724:84 , srv38999232:84 , srv400628360:84 , srv40277661:84 , srv407629705:84 , srv440242755:84 , srv475489485:84 , srv494195767:84 , srv511590:84 , srv529954068:84 , srv54896974:84 , srv579540481:84 , srv626845167:84 , srv658896340:84 , srv66956277:84 , srv681387762:84 , srv687370803:84 , srv691378868:84 , srv715641610:84 , srv724509459:84 , srv784709967:84 , srv806974588:84 , srv856756709:84 , srv89786528:84 , srv903560732:84 , srv951942719:84 , srv952905475:84 , srv976336190:84 , srv978053858:84 , srv983910889:84 , srv1072942886:85 , srv1206423623:85 , srv1260076443:85 , srv1532957398:85 , srv1672293043:85 , srv1718938300:85 , srv1757422996:85 , srv380628855:85 , srv457528471:85 , srv598449377:85 , srv780798524:684 } 2024-01-21 02:53:17,507 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=353 (was 340) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=12906 (was 13502) 2024-01-21 02:53:17,514 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=353, ProcessCount=168, AvailableMemoryMB=12906 2024-01-21 02:53:17,515 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-01-21 02:53:17,515 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-01-21 02:53:17,525 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1059072422:84 , srv1087984276:84 , srv1180535560:84 , srv122726697:84 , srv1383062737:84 , srv1429294488:84 , srv1556093418:84 , srv1623031803:84 , srv1643289946:84 , srv1651770621:84 , srv1704149674:84 , srv1745780834:84 , srv1799884811:84 , srv1814034531:84 , srv1902385355:84 , srv1935744636:84 , srv2073917594:84 , srv2094901360:84 , srv2108993232:84 , srv295502762:84 , srv304658716:84 , srv378432382:84 , srv490672565:84 , srv534380855:84 , srv634286203:84 , srv701924972:84 , srv752521383:84 , srv84890410:84 , srv920870289:84 , srv971698126:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-01-21 02:53:17,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-21 02:53:17,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1935744636=15, srv490672565=22, srv701924972=25, srv295502762=19, srv1180535560=2, srv634286203=24, srv1429294488=5, srv1704149674=10, srv304658716=20, srv1623031803=7, srv1902385355=14, srv1651770621=9, srv84890410=27, srv1556093418=6, srv1799884811=12, srv752521383=26, srv1383062737=4, srv122726697=3, srv920870289=28, srv971698126=29, srv1059072422=0, srv534380855=23, srv2108993232=18, srv1814034531=13, srv2094901360=17, srv378432382=21, srv1745780834=11, srv1087984276=1, srv1643289946=8, srv2073917594=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-21 02:53:17,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-21 02:53:17,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-21 02:53:17,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-21 02:53:17,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-21 02:53:17,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-21 02:53:17,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-21 02:53:17,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-21 02:53:17,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-21 02:53:17,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-21 02:53:17,551 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1022004565081, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-21 02:53:17,551 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912515073565584, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1022004565081, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-01-21 02:55:17,539 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6419182 different iterations. Found a solution that moves 1863 regions; Going from a computed imbalance of 0.9912515073565584 to a new imbalance of 4.3483349388262195E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.69, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-21 02:55:17,545 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1059072422:90 , srv1087984276:90 , srv1180535560:90 , srv122726697:90 , srv1383062737:90 , srv1429294488:90 , srv1556093418:90 , srv1623031803:90 , srv1643289946:90 , srv1651770621:90 , srv1704149674:90 , srv1745780834:90 , srv1799884811:90 , srv1814034531:90 , srv1902385355:90 , srv1935744636:90 , srv2073917594:90 , srv2094901360:90 , srv2108993232:90 , srv295502762:90 , srv304658716:90 , srv378432382:90 , srv490672565:90 , srv534380855:90 , srv634286203:90 , srv701924972:90 , srv752521383:90 , srv84890410:90 , srv920870289:90 , srv971698126:90 } 2024-01-21 02:55:17,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-01-21 02:55:17,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1935744636=15, srv490672565=22, srv701924972=25, srv295502762=19, srv1180535560=2, srv634286203=24, srv1429294488=5, srv1704149674=10, srv304658716=20, srv1623031803=7, srv1902385355=14, srv1651770621=9, srv84890410=27, srv1556093418=6, srv1799884811=12, srv752521383=26, srv1383062737=4, srv122726697=3, srv920870289=28, srv971698126=29, srv1059072422=0, srv534380855=23, srv2108993232=18, srv1814034531=13, srv2094901360=17, srv378432382=21, srv1745780834=11, srv1087984276=1, srv1643289946=8, srv2073917594=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-01-21 02:55:17,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-01-21 02:55:17,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-01-21 02:55:17,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-01-21 02:55:17,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-01-21 02:55:17,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-01-21 02:55:17,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-01-21 02:55:17,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-01-21 02:55:17,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-01-21 02:55:17,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-01-21 02:55:17,562 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-01-21 02:55:17,562 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1059072422:90 , srv1087984276:90 , srv1180535560:90 , srv122726697:90 , srv1383062737:90 , srv1429294488:90 , srv1556093418:90 , srv1623031803:90 , srv1643289946:90 , srv1651770621:90 , srv1704149674:90 , srv1745780834:90 , srv1799884811:90 , srv1814034531:90 , srv1902385355:90 , srv1935744636:90 , srv2073917594:90 , srv2094901360:90 , srv2108993232:90 , srv295502762:90 , srv304658716:90 , srv378432382:90 , srv490672565:90 , srv534380855:90 , srv634286203:90 , srv701924972:90 , srv752521383:90 , srv84890410:90 , srv920870289:90 , srv971698126:90 } 2024-01-21 02:55:17,576 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=275 (was 353), ProcessCount=167 (was 168), AvailableMemoryMB=11479 (was 12906)