2024-02-01 07:53:15,607 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-02-01 07:53:15,933 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-02-01 07:53:16,045 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-02-01 07:53:16,121 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-01 07:53:16,132 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-02-01 07:53:16,149 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=342, ProcessCount=167, AvailableMemoryMB=3157 2024-02-01 07:53:16,151 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-01 07:53:16,152 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-02-01 07:53:16,287 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1017590093:84 , srv1033245157:84 , srv1033543034:84 , srv1040143654:84 , srv1044559678:84 , srv1048272009:84 , srv1053555399:84 , srv1083302853:84 , srv1112918834:84 , srv1164039381:84 , srv1172430556:84 , srv1174781498:84 , srv118160894:84 , srv1189633921:84 , srv1203549228:84 , srv1227938807:84 , srv1262814074:84 , srv1282456099:84 , srv1285813226:84 , srv1328506219:84 , srv1380737646:84 , srv1381964707:84 , srv138664903:84 , srv1413520881:84 , srv1448861234:84 , srv1450355548:84 , srv1461607827:84 , srv1473776512:84 , srv1480388439:84 , srv1486992330:84 , srv1508719817:84 , srv1541162851:84 , srv1569098565:84 , srv1572617939:84 , srv160038721:84 , srv1604010656:84 , srv1645587575:84 , srv1651384852:84 , srv1700200236:84 , srv1708291354:84 , srv1714058901:84 , srv1810139530:84 , srv1810845605:84 , srv1836156695:84 , srv1859191754:84 , srv1861863857:84 , srv1927678427:84 , srv1960326712:84 , srv2000755868:84 , srv2098278327:84 , srv2126014065:84 , srv219013295:84 , srv222629332:84 , srv230550084:84 , srv284420689:84 , srv291958545:84 , srv325611896:84 , srv326908334:84 , srv328171049:84 , srv361398099:84 , srv39738085:84 , srv411397239:84 , srv438504492:84 , srv439563101:84 , srv457867310:84 , srv463216413:84 , srv479521580:84 , srv491158220:84 , srv51419152:84 , srv528525236:84 , srv55431077:84 , srv575971628:84 , srv576093696:84 , srv577800850:84 , srv584582396:84 , srv607079681:84 , srv642144651:84 , srv642398056:84 , srv653582534:84 , srv691440220:84 , srv696058656:84 , srv725333215:84 , srv736147560:84 , srv74242179:84 , srv760483958:84 , srv797205055:84 , srv803895867:84 , srv803930162:84 , srv824642547:84 , srv84126290:84 , srv851457053:84 , srv87547135:84 , srv890257061:84 , srv898607913:84 , srv923241401:84 , srv935838253:84 , srv937836378:84 , srv96792688:84 , srv971365053:84 , srv932587965:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-02-01 07:53:16,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-01 07:53:16,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv230550084=53, srv1927678427=46, srv1861863857=45, srv439563101=63, srv1541162851=31, srv1040143654=3, srv1044559678=4, srv1448861234=24, srv1450355548=25, srv851457053=90, srv890257061=92, srv1203549228=14, srv1700200236=38, srv584582396=74, srv1508719817=30, srv2098278327=49, srv1572617939=33, srv84126290=89, srv1960326712=47, srv457867310=64, srv2000755868=48, srv1708291354=39, srv491158220=67, srv935838253=96, srv55431077=70, srv1164039381=9, srv1033543034=2, srv1017590093=0, srv1651384852=37, srv326908334=57, srv160038721=34, srv1381964707=21, srv653582534=78, srv1859191754=44, srv803895867=86, srv577800850=73, srv1480388439=28, srv1053555399=6, srv2126014065=50, srv1569098565=32, srv1714058901=40, srv284420689=54, srv325611896=56, srv642144651=76, srv39738085=60, srv1033245157=1, srv51419152=68, srv118160894=12, srv1262814074=16, srv1083302853=7, srv725333215=81, srv1604010656=35, srv1328506219=19, srv87547135=91, srv1380737646=20, srv1413520881=23, srv528525236=69, srv1227938807=15, srv219013295=51, srv923241401=94, srv1645587575=36, srv1112918834=8, srv971365053=99, srv760483958=84, srv691440220=79, srv1473776512=27, srv932587965=95, srv797205055=85, srv1810845605=42, srv696058656=80, srv1048272009=5, srv576093696=72, srv438504492=62, srv1461607827=26, srv96792688=98, srv222629332=52, srv803930162=87, srv736147560=82, srv138664903=22, srv74242179=83, srv575971628=71, srv1172430556=10, srv291958545=55, srv328171049=58, srv463216413=65, srv937836378=97, srv1189633921=13, srv1174781498=11, srv361398099=59, srv607079681=75, srv1285813226=18, srv898607913=93, srv479521580=66, srv1282456099=17, srv1810139530=41, srv411397239=61, srv1836156695=43, srv824642547=88, srv642398056=77, srv1486992330=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-01 07:53:16,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-01 07:53:16,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-01 07:53:16,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-01 07:53:16,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-01 07:53:16,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-01 07:53:16,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-01 07:53:16,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-01 07:53:16,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-01 07:53:16,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-01 07:53:16,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-01 07:53:16,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-01 07:53:16,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-01 07:53:16,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-01 07:53:16,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-01 07:53:16,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-01 07:53:16,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-01 07:53:16,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-01 07:53:16,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-01 07:53:16,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-01 07:53:16,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-01 07:53:16,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-01 07:53:16,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-01 07:53:16,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-01 07:53:16,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-01 07:53:16,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-01 07:53:16,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-01 07:53:16,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-01 07:53:16,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-01 07:53:16,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-01 07:53:16,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-01 07:53:16,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-01 07:53:16,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-01 07:53:16,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-01 07:53:16,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-01 07:53:16,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-01 07:53:16,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-01 07:53:16,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-01 07:53:16,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-01 07:53:16,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-01 07:53:16,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-01 07:53:16,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-01 07:53:16,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-01 07:53:16,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-01 07:53:16,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-01 07:53:16,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-01 07:53:16,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-01 07:53:16,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-01 07:53:16,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-01 07:53:16,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-01 07:53:16,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-01 07:53:16,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-01 07:53:16,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-01 07:53:16,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-01 07:53:16,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-01 07:53:16,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-01 07:53:16,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-01 07:53:16,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-01 07:53:16,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-01 07:53:16,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-01 07:53:16,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-01 07:53:16,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-01 07:53:16,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-01 07:53:16,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-01 07:53:16,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-01 07:53:16,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-01 07:53:16,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-01 07:53:16,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-01 07:53:16,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-01 07:53:16,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-01 07:53:16,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-01 07:53:16,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-01 07:53:16,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-01 07:53:16,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-01 07:53:16,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-01 07:53:16,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-01 07:53:16,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-01 07:53:16,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-01 07:53:16,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-01 07:53:16,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-01 07:53:16,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-01 07:53:16,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-01 07:53:16,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-01 07:53:16,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-01 07:53:16,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-01 07:53:16,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-01 07:53:16,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-01 07:53:16,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-01 07:53:16,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-01 07:53:16,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-01 07:53:16,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-01 07:53:16,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-01 07:53:16,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-01 07:53:16,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-01 07:53:16,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-01 07:53:16,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-01 07:53:16,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-01 07:53:16,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-01 07:53:16,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-01 07:53:16,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-01 07:53:16,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-01 07:53:16,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-01 07:53:16,706 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 07:53:16,707 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-01 07:53:16,707 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-01 07:53:17,153 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 710 ms to try 5000 different iterations. Found a solution that moves 4713 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17568879764762488. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04182368469028587, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05955314332518656, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5236666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04182368469028587, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17091666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 07:53:17,175 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1282456099:81 , srv1033543034:83 , srv1569098565:83 , srv160038721:83 , srv1645587575:83 , srv2126014065:83 , srv230550084:83 , srv725333215:83 , srv935838253:83 , srv96792688:83 , srv1053555399:84 , srv1083302853:84 , srv1700200236:84 , srv1810139530:84 , srv1836156695:84 , srv291958545:84 , srv39738085:84 , srv438504492:84 , srv653582534:84 , srv803895867:84 , srv937836378:84 , srv1112918834:85 , srv1262814074:85 , srv1285813226:85 , srv1381964707:85 , srv1450355548:85 , srv1473776512:85 , srv1508719817:85 , srv1572617939:85 , srv1708291354:85 , srv1960326712:85 , srv2000755868:85 , srv328171049:85 , srv479521580:85 , srv51419152:85 , srv55431077:85 , srv696058656:85 , srv760483958:85 , srv797205055:85 , srv84126290:85 , srv1033245157:86 , srv1044559678:86 , srv1048272009:86 , srv1189633921:86 , srv1227938807:86 , srv1328506219:86 , srv1380737646:86 , srv138664903:86 , srv222629332:86 , srv325611896:86 , srv411397239:86 , srv439563101:86 , srv457867310:86 , srv576093696:86 , srv584582396:86 , srv824642547:86 , srv851457053:86 , srv923241401:86 , srv971365053:86 , srv1017590093:87 , srv1164039381:87 , srv1172430556:87 , srv1480388439:87 , srv1486992330:87 , srv1861863857:87 , srv326908334:87 , srv463216413:87 , srv491158220:87 , srv528525236:87 , srv575971628:87 , srv642144651:87 , srv642398056:87 , srv74242179:87 , srv898607913:87 , srv1448861234:88 , srv1461607827:88 , srv1604010656:88 , srv1651384852:88 , srv1714058901:88 , srv1859191754:88 , srv1927678427:88 , srv2098278327:88 , srv284420689:88 , srv361398099:88 , srv118160894:89 , srv691440220:89 , srv87547135:89 , srv1040143654:90 , srv1174781498:90 , srv1541162851:90 , srv1810845605:90 , srv219013295:90 , srv577800850:90 , srv736147560:90 , srv803930162:90 , srv890257061:90 , srv1203549228:91 , srv1413520881:91 , srv607079681:93 , srv932587965:462 } 2024-02-01 07:53:17,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-01 07:53:17,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv230550084=53, srv1927678427=46, srv1861863857=45, srv439563101=63, srv1541162851=31, srv1040143654=3, srv1044559678=4, srv1448861234=24, srv1450355548=25, srv851457053=90, srv890257061=92, srv1203549228=14, srv1700200236=38, srv584582396=74, srv1508719817=30, srv2098278327=49, srv1572617939=33, srv84126290=89, srv1960326712=47, srv457867310=64, srv2000755868=48, srv1708291354=39, srv491158220=67, srv935838253=96, srv55431077=70, srv1164039381=9, srv1033543034=2, srv1017590093=0, srv1651384852=37, srv326908334=57, srv160038721=34, srv1381964707=21, srv653582534=78, srv1859191754=44, srv803895867=86, srv577800850=73, srv1480388439=28, srv1053555399=6, srv2126014065=50, srv1569098565=32, srv1714058901=40, srv284420689=54, srv325611896=56, srv642144651=76, srv39738085=60, srv1033245157=1, srv51419152=68, srv118160894=12, srv1262814074=16, srv1083302853=7, srv725333215=81, srv1604010656=35, srv1328506219=19, srv87547135=91, srv1380737646=20, srv1413520881=23, srv528525236=69, srv1227938807=15, srv219013295=51, srv923241401=94, srv1645587575=36, srv1112918834=8, srv971365053=99, srv760483958=84, srv691440220=79, srv1473776512=27, srv932587965=95, srv797205055=85, srv1810845605=42, srv696058656=80, srv1048272009=5, srv576093696=72, srv438504492=62, srv1461607827=26, srv96792688=98, srv222629332=52, srv803930162=87, srv736147560=82, srv138664903=22, srv74242179=83, srv575971628=71, srv1172430556=10, srv291958545=55, srv328171049=58, srv463216413=65, srv937836378=97, srv1189633921=13, srv1174781498=11, srv361398099=59, srv607079681=75, srv1285813226=18, srv898607913=93, srv479521580=66, srv1282456099=17, srv1810139530=41, srv411397239=61, srv1836156695=43, srv824642547=88, srv642398056=77, srv1486992330=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-01 07:53:17,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-01 07:53:17,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-01 07:53:17,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-01 07:53:17,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-01 07:53:17,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-01 07:53:17,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-01 07:53:17,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-01 07:53:17,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-01 07:53:17,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-01 07:53:17,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-01 07:53:17,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-01 07:53:17,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-01 07:53:17,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-01 07:53:17,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-01 07:53:17,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-01 07:53:17,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-01 07:53:17,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-01 07:53:17,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-01 07:53:17,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-01 07:53:17,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-01 07:53:17,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-01 07:53:17,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-01 07:53:17,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-01 07:53:17,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-01 07:53:17,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-01 07:53:17,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-01 07:53:17,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-01 07:53:17,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-01 07:53:17,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-01 07:53:17,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-01 07:53:17,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-01 07:53:17,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-01 07:53:17,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-01 07:53:17,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-01 07:53:17,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-01 07:53:17,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-01 07:53:17,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-01 07:53:17,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-01 07:53:17,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-01 07:53:17,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-01 07:53:17,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-01 07:53:17,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-01 07:53:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-01 07:53:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-01 07:53:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-01 07:53:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-01 07:53:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-01 07:53:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-01 07:53:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-01 07:53:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-01 07:53:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-01 07:53:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-01 07:53:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-01 07:53:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-01 07:53:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-01 07:53:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-01 07:53:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-01 07:53:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-01 07:53:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-01 07:53:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-01 07:53:17,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-01 07:53:17,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-01 07:53:17,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-01 07:53:17,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-01 07:53:17,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-01 07:53:17,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-01 07:53:17,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-01 07:53:17,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-01 07:53:17,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-01 07:53:17,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-01 07:53:17,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-01 07:53:17,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-01 07:53:17,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-01 07:53:17,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-01 07:53:17,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-01 07:53:17,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-01 07:53:17,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-01 07:53:17,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-01 07:53:17,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-01 07:53:17,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-01 07:53:17,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-01 07:53:17,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-01 07:53:17,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-01 07:53:17,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-01 07:53:17,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-01 07:53:17,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-01 07:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-01 07:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-01 07:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-01 07:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-01 07:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-01 07:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-01 07:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-01 07:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-01 07:53:17,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-01 07:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-01 07:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-01 07:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-01 07:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-01 07:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-01 07:53:17,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-01 07:53:17,361 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04182368469028587, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05955314332518656, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04182368469028587, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17091666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 07:53:17,362 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-01 07:53:17,362 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17565579651627758, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04182368469028587, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05955314332518656, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04182368469028587, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17091666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-01 07:53:17,627 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 372 ms to try 5000 different iterations. Found a solution that moves 2579 regions; Going from a computed imbalance of 0.17565579651627758 to a new imbalance of 0.017382331731674288. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024413816134652943); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0330866291431805, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28655555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024413816134652943); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.014); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.049916666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 07:53:17,637 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1541162851:80 , srv219013295:80 , srv890257061:80 , srv118160894:81 , srv1203549228:81 , srv1413520881:81 , srv607079681:81 , srv803930162:81 , srv1174781498:82 , srv1461607827:82 , srv1714058901:82 , srv1810845605:82 , srv361398099:82 , srv691440220:82 , srv87547135:82 , srv1172430556:83 , srv1859191754:83 , srv2098278327:83 , srv284420689:83 , srv824642547:83 , srv923241401:83 , srv1227938807:84 , srv1380737646:84 , srv1486992330:84 , srv528525236:84 , srv575971628:84 , srv736147560:84 , srv1040143654:85 , srv1044559678:85 , srv1048272009:85 , srv1262814074:85 , srv138664903:85 , srv1448861234:85 , srv1473776512:85 , srv1604010656:85 , srv1651384852:85 , srv1861863857:85 , srv1960326712:85 , srv463216413:85 , srv577800850:85 , srv642144651:85 , srv898607913:85 , srv971365053:85 , srv1017590093:86 , srv1033245157:86 , srv1189633921:86 , srv1381964707:86 , srv1450355548:86 , srv1927678427:86 , srv2126014065:86 , srv222629332:86 , srv325611896:86 , srv411397239:86 , srv439563101:86 , srv457867310:86 , srv491158220:86 , srv642398056:86 , srv696058656:86 , srv797205055:86 , srv803895867:86 , srv851457053:86 , srv1033543034:87 , srv1053555399:87 , srv1164039381:87 , srv1285813226:87 , srv1480388439:87 , srv1508719817:87 , srv1569098565:87 , srv1572617939:87 , srv160038721:87 , srv1700200236:87 , srv1708291354:87 , srv1810139530:87 , srv291958545:87 , srv51419152:87 , srv584582396:87 , srv760483958:87 , srv96792688:87 , srv1112918834:88 , srv1836156695:88 , srv2000755868:88 , srv328171049:88 , srv39738085:88 , srv479521580:88 , srv576093696:88 , srv74242179:88 , srv935838253:88 , srv1083302853:89 , srv1282456099:89 , srv1328506219:89 , srv230550084:89 , srv438504492:89 , srv55431077:89 , srv653582534:89 , srv84126290:89 , srv937836378:89 , srv1645587575:90 , srv326908334:90 , srv725333215:90 , srv932587965:529 } 2024-02-01 07:53:17,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-01 07:53:17,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv230550084=53, srv1927678427=46, srv1861863857=45, srv439563101=63, srv1541162851=31, srv1040143654=3, srv1044559678=4, srv1448861234=24, srv1450355548=25, srv851457053=90, srv890257061=92, srv1203549228=14, srv1700200236=38, srv584582396=74, srv1508719817=30, srv2098278327=49, srv1572617939=33, srv84126290=89, srv1960326712=47, srv457867310=64, srv2000755868=48, srv1708291354=39, srv491158220=67, srv935838253=96, srv55431077=70, srv1164039381=9, srv1033543034=2, srv1017590093=0, srv1651384852=37, srv326908334=57, srv160038721=34, srv1381964707=21, srv653582534=78, srv1859191754=44, srv803895867=86, srv577800850=73, srv1480388439=28, srv1053555399=6, srv2126014065=50, srv1569098565=32, srv1714058901=40, srv284420689=54, srv325611896=56, srv642144651=76, srv39738085=60, srv1033245157=1, srv51419152=68, srv118160894=12, srv1262814074=16, srv1083302853=7, srv725333215=81, srv1604010656=35, srv1328506219=19, srv87547135=91, srv1380737646=20, srv1413520881=23, srv528525236=69, srv1227938807=15, srv219013295=51, srv923241401=94, srv1645587575=36, srv1112918834=8, srv971365053=99, srv760483958=84, srv691440220=79, srv1473776512=27, srv932587965=95, srv797205055=85, srv1810845605=42, srv696058656=80, srv1048272009=5, srv576093696=72, srv438504492=62, srv1461607827=26, srv96792688=98, srv222629332=52, srv803930162=87, srv736147560=82, srv138664903=22, srv74242179=83, srv575971628=71, srv1172430556=10, srv291958545=55, srv328171049=58, srv463216413=65, srv937836378=97, srv1189633921=13, srv1174781498=11, srv361398099=59, srv607079681=75, srv1285813226=18, srv898607913=93, srv479521580=66, srv1282456099=17, srv1810139530=41, srv411397239=61, srv1836156695=43, srv824642547=88, srv642398056=77, srv1486992330=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-01 07:53:17,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-01 07:53:17,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-01 07:53:17,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-01 07:53:17,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-01 07:53:17,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-01 07:53:17,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-01 07:53:17,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-01 07:53:17,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-01 07:53:17,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-01 07:53:17,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-01 07:53:17,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-01 07:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-01 07:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-01 07:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-01 07:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-01 07:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-01 07:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-01 07:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-01 07:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-01 07:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-01 07:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-01 07:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-01 07:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-01 07:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-01 07:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-01 07:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-01 07:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-01 07:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-01 07:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-01 07:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-01 07:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-01 07:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-01 07:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-01 07:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-01 07:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-01 07:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-01 07:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-01 07:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-01 07:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-01 07:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-01 07:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-01 07:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-01 07:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-01 07:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-01 07:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-01 07:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-01 07:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-01 07:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-01 07:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-01 07:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-01 07:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-01 07:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-01 07:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-01 07:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-01 07:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-01 07:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-01 07:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-01 07:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-01 07:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-01 07:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-01 07:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-01 07:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-01 07:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-01 07:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-01 07:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-01 07:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-01 07:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-01 07:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-01 07:53:17,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-01 07:53:17,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-01 07:53:17,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-01 07:53:17,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-01 07:53:17,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-01 07:53:17,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-01 07:53:17,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-01 07:53:17,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-01 07:53:17,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-01 07:53:17,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-01 07:53:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-01 07:53:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-01 07:53:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-01 07:53:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-01 07:53:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-01 07:53:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-01 07:53:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-01 07:53:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-01 07:53:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-01 07:53:17,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-01 07:53:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-01 07:53:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-01 07:53:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-01 07:53:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-01 07:53:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-01 07:53:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-01 07:53:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-01 07:53:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-01 07:53:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-01 07:53:17,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-01 07:53:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-01 07:53:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-01 07:53:17,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-01 07:53:17,795 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024413816134652943); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0330866291431805, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024413816134652943); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.014); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.049916666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 07:53:17,796 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-01 07:53:17,796 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.017364273187701288, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024413816134652943); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0330866291431805, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024413816134652943); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.014); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.049916666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-01 07:53:18,088 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 389 ms to try 5000 different iterations. Found a solution that moves 1005 regions; Going from a computed imbalance of 0.017364273187701288 to a new imbalance of 0.0014272206765473597. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009022563891956483); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010845305479975394); MoveCostFunction : (multiplier=7.0, imbalance=0.11166666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.009022563891956483); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=9.166666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 07:53:18,090 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv326908334:79 , srv1328506219:81 , srv1017590093:83 , srv1112918834:83 , srv1164039381:83 , srv491158220:83 , srv55431077:83 , srv576093696:83 , srv577800850:83 , srv736147560:83 , srv74242179:83 , srv84126290:83 , srv1033245157:84 , srv1040143654:84 , srv1083302853:84 , srv1448861234:84 , srv1480388439:84 , srv1508719817:84 , srv1604010656:84 , srv1645587575:84 , srv1861863857:84 , srv328171049:84 , srv438504492:84 , srv479521580:84 , srv51419152:84 , srv607079681:84 , srv653582534:84 , srv1189633921:85 , srv1285813226:85 , srv1413520881:85 , srv1572617939:85 , srv1651384852:85 , srv1708291354:85 , srv1810139530:85 , srv1810845605:85 , srv1836156695:85 , srv1927678427:85 , srv2000755868:85 , srv230550084:85 , srv291958545:85 , srv325611896:85 , srv39738085:85 , srv411397239:85 , srv439563101:85 , srv584582396:85 , srv642398056:85 , srv725333215:85 , srv760483958:85 , srv851457053:85 , srv898607913:85 , srv935838253:85 , srv937836378:85 , srv1033543034:86 , srv1044559678:86 , srv1053555399:86 , srv1174781498:86 , srv1227938807:86 , srv1282456099:86 , srv1380737646:86 , srv1450355548:86 , srv1473776512:86 , srv1486992330:86 , srv1569098565:86 , srv1700200236:86 , srv1859191754:86 , srv284420689:86 , srv457867310:86 , srv463216413:86 , srv642144651:86 , srv696058656:86 , srv87547135:86 , srv890257061:86 , srv971365053:86 , srv1048272009:87 , srv118160894:87 , srv1203549228:87 , srv138664903:87 , srv1461607827:87 , srv1541162851:87 , srv160038721:87 , srv1960326712:87 , srv2098278327:87 , srv219013295:87 , srv222629332:87 , srv528525236:87 , srv575971628:87 , srv691440220:87 , srv797205055:87 , srv803895867:87 , srv803930162:87 , srv96792688:87 , srv1172430556:88 , srv1262814074:88 , srv1381964707:88 , srv1714058901:88 , srv361398099:88 , srv923241401:88 , srv2126014065:89 , srv824642547:89 , srv932587965:547 } 2024-02-01 07:53:18,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-01 07:53:18,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv230550084=53, srv1927678427=46, srv1861863857=45, srv439563101=63, srv1541162851=31, srv1040143654=3, srv1044559678=4, srv1448861234=24, srv1450355548=25, srv851457053=90, srv890257061=92, srv1203549228=14, srv1700200236=38, srv584582396=74, srv1508719817=30, srv2098278327=49, srv1572617939=33, srv84126290=89, srv1960326712=47, srv457867310=64, srv2000755868=48, srv1708291354=39, srv491158220=67, srv935838253=96, srv55431077=70, srv1164039381=9, srv1033543034=2, srv1017590093=0, srv1651384852=37, srv326908334=57, srv160038721=34, srv1381964707=21, srv653582534=78, srv1859191754=44, srv803895867=86, srv577800850=73, srv1480388439=28, srv1053555399=6, srv2126014065=50, srv1569098565=32, srv1714058901=40, srv284420689=54, srv325611896=56, srv642144651=76, srv39738085=60, srv1033245157=1, srv51419152=68, srv118160894=12, srv1262814074=16, srv1083302853=7, srv725333215=81, srv1604010656=35, srv1328506219=19, srv87547135=91, srv1380737646=20, srv1413520881=23, srv528525236=69, srv1227938807=15, srv219013295=51, srv923241401=94, srv1645587575=36, srv1112918834=8, srv971365053=99, srv760483958=84, srv691440220=79, srv1473776512=27, srv932587965=95, srv797205055=85, srv1810845605=42, srv696058656=80, srv1048272009=5, srv576093696=72, srv438504492=62, srv1461607827=26, srv96792688=98, srv222629332=52, srv803930162=87, srv736147560=82, srv138664903=22, srv74242179=83, srv575971628=71, srv1172430556=10, srv291958545=55, srv328171049=58, srv463216413=65, srv937836378=97, srv1189633921=13, srv1174781498=11, srv361398099=59, srv607079681=75, srv1285813226=18, srv898607913=93, srv479521580=66, srv1282456099=17, srv1810139530=41, srv411397239=61, srv1836156695=43, srv824642547=88, srv642398056=77, srv1486992330=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-01 07:53:18,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-01 07:53:18,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-01 07:53:18,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-01 07:53:18,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-01 07:53:18,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-01 07:53:18,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-01 07:53:18,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-01 07:53:18,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-01 07:53:18,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-01 07:53:18,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-01 07:53:18,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-01 07:53:18,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-01 07:53:18,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-01 07:53:18,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-01 07:53:18,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-01 07:53:18,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-01 07:53:18,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-01 07:53:18,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-01 07:53:18,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-01 07:53:18,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-01 07:53:18,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-01 07:53:18,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-01 07:53:18,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-01 07:53:18,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-01 07:53:18,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-01 07:53:18,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-01 07:53:18,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-01 07:53:18,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-01 07:53:18,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-01 07:53:18,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-01 07:53:18,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-01 07:53:18,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-01 07:53:18,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-01 07:53:18,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-01 07:53:18,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-01 07:53:18,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-01 07:53:18,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-01 07:53:18,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-01 07:53:18,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-01 07:53:18,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-01 07:53:18,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-01 07:53:18,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-01 07:53:18,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-01 07:53:18,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-01 07:53:18,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-01 07:53:18,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-01 07:53:18,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-01 07:53:18,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-01 07:53:18,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-01 07:53:18,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-01 07:53:18,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-01 07:53:18,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-01 07:53:18,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-01 07:53:18,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-01 07:53:18,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-01 07:53:18,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-01 07:53:18,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-01 07:53:18,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-01 07:53:18,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-01 07:53:18,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-01 07:53:18,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-01 07:53:18,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-01 07:53:18,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-01 07:53:18,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-01 07:53:18,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-01 07:53:18,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-01 07:53:18,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-01 07:53:18,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-01 07:53:18,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-01 07:53:18,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-01 07:53:18,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-01 07:53:18,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-01 07:53:18,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-01 07:53:18,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-01 07:53:18,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-01 07:53:18,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-01 07:53:18,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-01 07:53:18,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-01 07:53:18,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-01 07:53:18,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-01 07:53:18,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-01 07:53:18,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-01 07:53:18,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-01 07:53:18,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-01 07:53:18,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-01 07:53:18,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-01 07:53:18,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-01 07:53:18,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-01 07:53:18,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-01 07:53:18,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-01 07:53:18,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-01 07:53:18,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-01 07:53:18,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-01 07:53:18,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-01 07:53:18,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-01 07:53:18,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-01 07:53:18,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-01 07:53:18,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-01 07:53:18,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-01 07:53:18,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-01 07:53:18,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-01 07:53:18,242 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009022563891956483); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010845305479975394); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.009022563891956483); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=9.166666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 07:53:18,242 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-01 07:53:18,242 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0014201835161391145, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009022563891956483); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010845305479975394); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.009022563891956483); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=9.166666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-01 07:53:18,656 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 521 ms to try 5000 different iterations. Found a solution that moves 323 regions; Going from a computed imbalance of 0.0014201835161391145 to a new imbalance of 7.567739327605994E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024158102511315956); MoveCostFunction : (multiplier=7.0, imbalance=0.03588888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=6.666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 07:53:18,657 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1040143654:83 , srv1381964707:83 , srv1927678427:83 , srv222629332:83 , srv361398099:83 , srv607079681:83 , srv1048272009:84 , srv1172430556:84 , srv1174781498:84 , srv1203549228:84 , srv1262814074:84 , srv138664903:84 , srv1480388439:84 , srv1541162851:84 , srv1651384852:84 , srv1714058901:84 , srv1960326712:84 , srv2000755868:84 , srv2126014065:84 , srv219013295:84 , srv439563101:84 , srv457867310:84 , srv463216413:84 , srv528525236:84 , srv575971628:84 , srv584582396:84 , srv642144651:84 , srv642398056:84 , srv691440220:84 , srv725333215:84 , srv797205055:84 , srv803930162:84 , srv824642547:84 , srv937836378:84 , srv1044559678:85 , srv1053555399:85 , srv1083302853:85 , srv1164039381:85 , srv118160894:85 , srv1189633921:85 , srv1227938807:85 , srv1285813226:85 , srv1328506219:85 , srv1413520881:85 , srv1450355548:85 , srv1461607827:85 , srv1486992330:85 , srv1572617939:85 , srv160038721:85 , srv1604010656:85 , srv1645587575:85 , srv1700200236:85 , srv1708291354:85 , srv1810845605:85 , srv1859191754:85 , srv2098278327:85 , srv230550084:85 , srv284420689:85 , srv291958545:85 , srv325611896:85 , srv328171049:85 , srv39738085:85 , srv411397239:85 , srv479521580:85 , srv55431077:85 , srv576093696:85 , srv577800850:85 , srv653582534:85 , srv696058656:85 , srv736147560:85 , srv74242179:85 , srv760483958:85 , srv803895867:85 , srv84126290:85 , srv87547135:85 , srv890257061:85 , srv898607913:85 , srv923241401:85 , srv935838253:85 , srv96792688:85 , srv971365053:85 , srv1033245157:86 , srv1033543034:86 , srv1112918834:86 , srv1282456099:86 , srv1380737646:86 , srv1448861234:86 , srv1473776512:86 , srv1508719817:86 , srv1569098565:86 , srv1810139530:86 , srv1836156695:86 , srv326908334:86 , srv438504492:86 , srv491158220:86 , srv51419152:86 , srv851457053:86 , srv1017590093:87 , srv1861863857:87 , srv932587965:605 } 2024-02-01 07:53:18,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-01 07:53:18,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv230550084=53, srv1927678427=46, srv1861863857=45, srv439563101=63, srv1541162851=31, srv1040143654=3, srv1044559678=4, srv1448861234=24, srv1450355548=25, srv851457053=90, srv890257061=92, srv1203549228=14, srv1700200236=38, srv584582396=74, srv1508719817=30, srv2098278327=49, srv1572617939=33, srv84126290=89, srv1960326712=47, srv457867310=64, srv2000755868=48, srv1708291354=39, srv491158220=67, srv935838253=96, srv55431077=70, srv1164039381=9, srv1033543034=2, srv1017590093=0, srv1651384852=37, srv326908334=57, srv160038721=34, srv1381964707=21, srv653582534=78, srv1859191754=44, srv803895867=86, srv577800850=73, srv1480388439=28, srv1053555399=6, srv2126014065=50, srv1569098565=32, srv1714058901=40, srv284420689=54, srv325611896=56, srv642144651=76, srv39738085=60, srv1033245157=1, srv51419152=68, srv118160894=12, srv1262814074=16, srv1083302853=7, srv725333215=81, srv1604010656=35, srv1328506219=19, srv87547135=91, srv1380737646=20, srv1413520881=23, srv528525236=69, srv1227938807=15, srv219013295=51, srv923241401=94, srv1645587575=36, srv1112918834=8, srv971365053=99, srv760483958=84, srv691440220=79, srv1473776512=27, srv932587965=95, srv797205055=85, srv1810845605=42, srv696058656=80, srv1048272009=5, srv576093696=72, srv438504492=62, srv1461607827=26, srv96792688=98, srv222629332=52, srv803930162=87, srv736147560=82, srv138664903=22, srv74242179=83, srv575971628=71, srv1172430556=10, srv291958545=55, srv328171049=58, srv463216413=65, srv937836378=97, srv1189633921=13, srv1174781498=11, srv361398099=59, srv607079681=75, srv1285813226=18, srv898607913=93, srv479521580=66, srv1282456099=17, srv1810139530=41, srv411397239=61, srv1836156695=43, srv824642547=88, srv642398056=77, srv1486992330=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-01 07:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-01 07:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-01 07:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-01 07:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-01 07:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-01 07:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-01 07:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-01 07:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-01 07:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-01 07:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-01 07:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-01 07:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-01 07:53:18,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-01 07:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-01 07:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-01 07:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-01 07:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-01 07:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-01 07:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-01 07:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-01 07:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-01 07:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-01 07:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-01 07:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-01 07:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-01 07:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-01 07:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-01 07:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-01 07:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-01 07:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-01 07:53:18,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-01 07:53:18,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-01 07:53:18,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-01 07:53:18,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-01 07:53:18,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-01 07:53:18,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-01 07:53:18,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-01 07:53:18,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-01 07:53:18,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-01 07:53:18,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-01 07:53:18,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-01 07:53:18,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-01 07:53:18,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-01 07:53:18,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-01 07:53:18,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-01 07:53:18,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-01 07:53:18,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-01 07:53:18,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-01 07:53:18,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-01 07:53:18,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-01 07:53:18,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-01 07:53:18,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-01 07:53:18,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-01 07:53:18,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-01 07:53:18,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-01 07:53:18,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-01 07:53:18,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-01 07:53:18,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-01 07:53:18,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-01 07:53:18,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-01 07:53:18,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-01 07:53:18,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-01 07:53:18,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-01 07:53:18,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-01 07:53:18,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-01 07:53:18,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-01 07:53:18,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-01 07:53:18,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-01 07:53:18,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-01 07:53:18,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-01 07:53:18,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-01 07:53:18,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-01 07:53:18,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-01 07:53:18,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-01 07:53:18,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-01 07:53:18,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-01 07:53:18,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-01 07:53:18,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-01 07:53:18,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-01 07:53:18,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-01 07:53:18,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-01 07:53:18,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-01 07:53:18,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-01 07:53:18,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-01 07:53:18,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-01 07:53:18,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-01 07:53:18,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-01 07:53:18,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-01 07:53:18,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-01 07:53:18,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-01 07:53:18,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-01 07:53:18,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-01 07:53:18,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-01 07:53:18,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-01 07:53:18,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-01 07:53:18,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-01 07:53:18,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-01 07:53:18,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-01 07:53:18,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-01 07:53:18,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-01 07:53:18,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-01 07:53:18,798 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024158102511315956); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=6.666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 07:53:18,799 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-01 07:53:18,799 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.341569893589752E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024158102511315956); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=6.666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-01 07:53:19,087 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 397 ms to try 5000 different iterations. Found a solution that moves 51 regions; Going from a computed imbalance of 7.341569893589752E-5 to a new imbalance of 8.046531433008006E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.005666666666666667); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 07:53:19,088 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1017590093:83 , srv1203549228:83 , srv1448861234:83 , srv1836156695:83 , srv1861863857:83 , srv2098278327:83 , srv438504492:83 , srv577800850:83 , srv74242179:83 , srv851457053:83 , srv1033245157:84 , srv1033543034:84 , srv1040143654:84 , srv1044559678:84 , srv1048272009:84 , srv1053555399:84 , srv1083302853:84 , srv1112918834:84 , srv1164039381:84 , srv1172430556:84 , srv1174781498:84 , srv118160894:84 , srv1189633921:84 , srv1262814074:84 , srv1282456099:84 , srv1285813226:84 , srv1380737646:84 , srv1381964707:84 , srv138664903:84 , srv1413520881:84 , srv1450355548:84 , srv1461607827:84 , srv1473776512:84 , srv1480388439:84 , srv1486992330:84 , srv1508719817:84 , srv1569098565:84 , srv1572617939:84 , srv160038721:84 , srv1604010656:84 , srv1645587575:84 , srv1651384852:84 , srv1700200236:84 , srv1708291354:84 , srv1714058901:84 , srv1810139530:84 , srv1810845605:84 , srv1859191754:84 , srv1927678427:84 , srv2000755868:84 , srv2126014065:84 , srv222629332:84 , srv230550084:84 , srv284420689:84 , srv325611896:84 , srv326908334:84 , srv328171049:84 , srv39738085:84 , srv411397239:84 , srv457867310:84 , srv463216413:84 , srv479521580:84 , srv491158220:84 , srv51419152:84 , srv528525236:84 , srv55431077:84 , srv575971628:84 , srv576093696:84 , srv584582396:84 , srv642144651:84 , srv642398056:84 , srv653582534:84 , srv691440220:84 , srv696058656:84 , srv725333215:84 , srv736147560:84 , srv760483958:84 , srv797205055:84 , srv803895867:84 , srv803930162:84 , srv824642547:84 , srv84126290:84 , srv87547135:84 , srv898607913:84 , srv923241401:84 , srv937836378:84 , srv96792688:84 , srv971365053:84 , srv1227938807:85 , srv1328506219:85 , srv1541162851:85 , srv1960326712:85 , srv219013295:85 , srv291958545:85 , srv361398099:85 , srv439563101:85 , srv607079681:85 , srv890257061:85 , srv935838253:85 , srv932587965:683 } 2024-02-01 07:53:19,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-01 07:53:19,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv230550084=53, srv1927678427=46, srv1861863857=45, srv439563101=63, srv1541162851=31, srv1040143654=3, srv1044559678=4, srv1448861234=24, srv1450355548=25, srv851457053=90, srv890257061=92, srv1203549228=14, srv1700200236=38, srv584582396=74, srv1508719817=30, srv2098278327=49, srv1572617939=33, srv84126290=89, srv1960326712=47, srv457867310=64, srv2000755868=48, srv1708291354=39, srv491158220=67, srv935838253=96, srv55431077=70, srv1164039381=9, srv1033543034=2, srv1017590093=0, srv1651384852=37, srv326908334=57, srv160038721=34, srv1381964707=21, srv653582534=78, srv1859191754=44, srv803895867=86, srv577800850=73, srv1480388439=28, srv1053555399=6, srv2126014065=50, srv1569098565=32, srv1714058901=40, srv284420689=54, srv325611896=56, srv642144651=76, srv39738085=60, srv1033245157=1, srv51419152=68, srv118160894=12, srv1262814074=16, srv1083302853=7, srv725333215=81, srv1604010656=35, srv1328506219=19, srv87547135=91, srv1380737646=20, srv1413520881=23, srv528525236=69, srv1227938807=15, srv219013295=51, srv923241401=94, srv1645587575=36, srv1112918834=8, srv971365053=99, srv760483958=84, srv691440220=79, srv1473776512=27, srv932587965=95, srv797205055=85, srv1810845605=42, srv696058656=80, srv1048272009=5, srv576093696=72, srv438504492=62, srv1461607827=26, srv96792688=98, srv222629332=52, srv803930162=87, srv736147560=82, srv138664903=22, srv74242179=83, srv575971628=71, srv1172430556=10, srv291958545=55, srv328171049=58, srv463216413=65, srv937836378=97, srv1189633921=13, srv1174781498=11, srv361398099=59, srv607079681=75, srv1285813226=18, srv898607913=93, srv479521580=66, srv1282456099=17, srv1810139530=41, srv411397239=61, srv1836156695=43, srv824642547=88, srv642398056=77, srv1486992330=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-01 07:53:19,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-01 07:53:19,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-01 07:53:19,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-01 07:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-01 07:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-01 07:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-01 07:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-01 07:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-01 07:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-01 07:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-01 07:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-01 07:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-01 07:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-01 07:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-01 07:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-01 07:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-01 07:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-01 07:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-01 07:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-01 07:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-01 07:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-01 07:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-01 07:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-01 07:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-01 07:53:19,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-01 07:53:19,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-01 07:53:19,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-01 07:53:19,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-01 07:53:19,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-01 07:53:19,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-01 07:53:19,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-01 07:53:19,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-01 07:53:19,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-01 07:53:19,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-01 07:53:19,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-01 07:53:19,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-01 07:53:19,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-01 07:53:19,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-01 07:53:19,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-01 07:53:19,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-01 07:53:19,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-01 07:53:19,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-01 07:53:19,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-01 07:53:19,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-01 07:53:19,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-01 07:53:19,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-01 07:53:19,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-01 07:53:19,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-01 07:53:19,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-01 07:53:19,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-01 07:53:19,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-01 07:53:19,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-01 07:53:19,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-01 07:53:19,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-01 07:53:19,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-01 07:53:19,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-01 07:53:19,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-01 07:53:19,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-01 07:53:19,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-01 07:53:19,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-01 07:53:19,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-01 07:53:19,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-01 07:53:19,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-01 07:53:19,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-01 07:53:19,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-01 07:53:19,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-01 07:53:19,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-01 07:53:19,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-01 07:53:19,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-01 07:53:19,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-01 07:53:19,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-01 07:53:19,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-01 07:53:19,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-01 07:53:19,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-01 07:53:19,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-01 07:53:19,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-01 07:53:19,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-01 07:53:19,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-01 07:53:19,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-01 07:53:19,239 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.689421800350782E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 07:53:19,239 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1017590093:83 , srv1203549228:83 , srv1448861234:83 , srv1836156695:83 , srv1861863857:83 , srv2098278327:83 , srv438504492:83 , srv577800850:83 , srv74242179:83 , srv851457053:83 , srv1033245157:84 , srv1033543034:84 , srv1040143654:84 , srv1044559678:84 , srv1048272009:84 , srv1053555399:84 , srv1083302853:84 , srv1112918834:84 , srv1164039381:84 , srv1172430556:84 , srv1174781498:84 , srv118160894:84 , srv1189633921:84 , srv1262814074:84 , srv1282456099:84 , srv1285813226:84 , srv1380737646:84 , srv1381964707:84 , srv138664903:84 , srv1413520881:84 , srv1450355548:84 , srv1461607827:84 , srv1473776512:84 , srv1480388439:84 , srv1486992330:84 , srv1508719817:84 , srv1569098565:84 , srv1572617939:84 , srv160038721:84 , srv1604010656:84 , srv1645587575:84 , srv1651384852:84 , srv1700200236:84 , srv1708291354:84 , srv1714058901:84 , srv1810139530:84 , srv1810845605:84 , srv1859191754:84 , srv1927678427:84 , srv2000755868:84 , srv2126014065:84 , srv222629332:84 , srv230550084:84 , srv284420689:84 , srv325611896:84 , srv326908334:84 , srv328171049:84 , srv39738085:84 , srv411397239:84 , srv457867310:84 , srv463216413:84 , srv479521580:84 , srv491158220:84 , srv51419152:84 , srv528525236:84 , srv55431077:84 , srv575971628:84 , srv576093696:84 , srv584582396:84 , srv642144651:84 , srv642398056:84 , srv653582534:84 , srv691440220:84 , srv696058656:84 , srv725333215:84 , srv736147560:84 , srv760483958:84 , srv797205055:84 , srv803895867:84 , srv803930162:84 , srv824642547:84 , srv84126290:84 , srv87547135:84 , srv898607913:84 , srv923241401:84 , srv937836378:84 , srv96792688:84 , srv971365053:84 , srv1227938807:85 , srv1328506219:85 , srv1541162851:85 , srv1960326712:85 , srv219013295:85 , srv291958545:85 , srv361398099:85 , srv439563101:85 , srv607079681:85 , srv890257061:85 , srv935838253:85 , srv932587965:683 } 2024-02-01 07:53:19,269 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=342 (was 342), ProcessCount=167 (was 167), AvailableMemoryMB=2267 (was 3157) 2024-02-01 07:53:19,278 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=342, ProcessCount=167, AvailableMemoryMB=2267 2024-02-01 07:53:19,279 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-01 07:53:19,279 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-02-01 07:53:19,293 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1020383668:84 , srv108641087:84 , srv1173787771:84 , srv1215639810:84 , srv1282626971:84 , srv1289090059:84 , srv1331838898:84 , srv1356700850:84 , srv1401629600:84 , srv1487285264:84 , srv1496890139:84 , srv1542032790:84 , srv1789762827:84 , srv1815624938:84 , srv1949230683:84 , srv2015490057:84 , srv2027738241:84 , srv2135354601:84 , srv2143987838:84 , srv238670641:84 , srv419541978:84 , srv494635277:84 , srv574340559:84 , srv69678983:84 , srv708007445:84 , srv804086494:84 , srv873792723:84 , srv878962355:84 , srv985739297:84 , srv892141589:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-02-01 07:53:19,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-01 07:53:19,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1020383668=0, srv1815624938=13, srv1949230683=14, srv2015490057=15, srv878962355=27, srv419541978=20, srv1496890139=10, srv574340559=22, srv804086494=25, srv2027738241=16, srv1401629600=8, srv1215639810=3, srv985739297=29, srv1789762827=12, srv238670641=19, srv2135354601=17, srv1173787771=2, srv1356700850=7, srv892141589=28, srv1542032790=11, srv1331838898=6, srv708007445=24, srv1487285264=9, srv494635277=21, srv1282626971=4, srv873792723=26, srv69678983=23, srv1289090059=5, srv108641087=1, srv2143987838=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-01 07:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-01 07:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-01 07:53:19,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-01 07:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-01 07:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-01 07:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-01 07:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-01 07:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-01 07:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-01 07:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-01 07:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-01 07:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-01 07:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-01 07:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-01 07:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-01 07:53:19,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-01 07:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-01 07:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-01 07:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-01 07:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-01 07:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-01 07:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-01 07:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-01 07:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-01 07:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-01 07:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-01 07:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-01 07:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-01 07:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-01 07:53:19,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-01 07:53:19,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-02-01 07:53:19,332 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1205556100314633, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 07:53:19,333 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912572910054986, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1205556100314633, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-02-01 07:55:19,315 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6070935 different iterations. Found a solution that moves 1848 regions; Going from a computed imbalance of 0.9912572910054986 to a new imbalance of 4.313324190526491E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6844444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 07:55:19,320 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1020383668:90 , srv108641087:90 , srv1173787771:90 , srv1215639810:90 , srv1282626971:90 , srv1289090059:90 , srv1331838898:90 , srv1356700850:90 , srv1401629600:90 , srv1487285264:90 , srv1496890139:90 , srv1542032790:90 , srv1789762827:90 , srv1815624938:90 , srv1949230683:90 , srv2015490057:90 , srv2027738241:90 , srv2135354601:90 , srv2143987838:90 , srv238670641:90 , srv419541978:90 , srv494635277:90 , srv574340559:90 , srv69678983:90 , srv708007445:90 , srv804086494:90 , srv873792723:90 , srv878962355:90 , srv892141589:90 , srv985739297:90 } 2024-02-01 07:55:19,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-01 07:55:19,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1020383668=0, srv1815624938=13, srv1949230683=14, srv2015490057=15, srv878962355=27, srv419541978=20, srv1496890139=10, srv574340559=22, srv804086494=25, srv2027738241=16, srv1401629600=8, srv1215639810=3, srv985739297=29, srv1789762827=12, srv238670641=19, srv2135354601=17, srv1173787771=2, srv1356700850=7, srv892141589=28, srv1542032790=11, srv1331838898=6, srv708007445=24, srv1487285264=9, srv494635277=21, srv1282626971=4, srv873792723=26, srv69678983=23, srv1289090059=5, srv108641087=1, srv2143987838=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-01 07:55:19,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-01 07:55:19,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-01 07:55:19,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-01 07:55:19,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-01 07:55:19,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-01 07:55:19,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-01 07:55:19,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-01 07:55:19,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-01 07:55:19,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-01 07:55:19,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-01 07:55:19,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-01 07:55:19,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-01 07:55:19,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-01 07:55:19,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-01 07:55:19,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-01 07:55:19,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-01 07:55:19,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-01 07:55:19,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-01 07:55:19,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-01 07:55:19,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-01 07:55:19,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-01 07:55:19,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-01 07:55:19,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-01 07:55:19,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-01 07:55:19,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-01 07:55:19,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-01 07:55:19,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-01 07:55:19,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-01 07:55:19,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-01 07:55:19,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-01 07:55:19,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-02-01 07:55:19,339 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-01 07:55:19,339 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1020383668:90 , srv108641087:90 , srv1173787771:90 , srv1215639810:90 , srv1282626971:90 , srv1289090059:90 , srv1331838898:90 , srv1356700850:90 , srv1401629600:90 , srv1487285264:90 , srv1496890139:90 , srv1542032790:90 , srv1789762827:90 , srv1815624938:90 , srv1949230683:90 , srv2015490057:90 , srv2027738241:90 , srv2135354601:90 , srv2143987838:90 , srv238670641:90 , srv419541978:90 , srv494635277:90 , srv574340559:90 , srv69678983:90 , srv708007445:90 , srv804086494:90 , srv873792723:90 , srv878962355:90 , srv892141589:90 , srv985739297:90 } 2024-02-01 07:55:19,353 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=277 (was 342), ProcessCount=167 (was 167), AvailableMemoryMB=2438 (was 2267) - AvailableMemoryMB LEAK? -