2023-07-23 16:53:12,260 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-07-23 16:53:12,573 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-23 16:53:12,694 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-23 16:53:12,770 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-23 16:53:12,780 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-23 16:53:12,795 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=241, ProcessCount=175, AvailableMemoryMB=10561 2023-07-23 16:53:12,797 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-23 16:53:12,797 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-23 16:53:12,957 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1052288810:84 , srv1073671122:84 , srv1076165402:84 , srv1112568363:84 , srv1112946066:84 , srv1142072580:84 , srv118004462:84 , srv1194052408:84 , srv1200642332:84 , srv1209738172:84 , srv1225149006:84 , srv1230618723:84 , srv123675539:84 , srv1249084800:84 , srv1261538529:84 , srv1279482853:84 , srv1310248395:84 , srv1312488177:84 , srv1331558790:84 , srv1331924925:84 , srv1357250365:84 , srv1421582046:84 , srv1427095674:84 , srv1446456836:84 , srv1472158368:84 , srv1486190803:84 , srv1574160707:84 , srv1619807643:84 , srv1684827893:84 , srv170121053:84 , srv1714537929:84 , srv1716466189:84 , srv1732616078:84 , srv173468065:84 , srv1770990673:84 , srv1772841098:84 , srv1772912748:84 , srv1793706794:84 , srv1793878669:84 , srv1794875250:84 , srv1808080303:84 , srv1810559156:84 , srv1842205637:84 , srv1865212614:84 , srv1888127565:84 , srv1917173776:84 , srv1922493806:84 , srv194963036:84 , srv1952714031:84 , srv1957939575:84 , srv1976113794:84 , srv198874522:84 , srv1999568051:84 , srv200479936:84 , srv2015270801:84 , srv2020447734:84 , srv2055210181:84 , srv2093848918:84 , srv2105350864:84 , srv2114755469:84 , srv237768655:84 , srv252292607:84 , srv305991306:84 , srv306807008:84 , srv345735342:84 , srv353177431:84 , srv361420310:84 , srv3616107:84 , srv370340639:84 , srv371875736:84 , srv417316523:84 , srv423197526:84 , srv435743224:84 , srv442789462:84 , srv465445978:84 , srv518482735:84 , srv524157225:84 , srv52637337:84 , srv553451783:84 , srv559839211:84 , srv600203098:84 , srv649988250:84 , srv656757113:84 , srv670774647:84 , srv688522531:84 , srv725906170:84 , srv757363856:84 , srv787228314:84 , srv806734757:84 , srv88351242:84 , srv896129139:84 , srv903132673:84 , srv910161676:84 , srv913223977:84 , srv929107684:84 , srv938423023:84 , srv969359493:84 , srv98152467:84 , srv992951381:84 , srv268863680:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-07-23 16:53:12,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-23 16:53:12,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv353177431=66, srv1574160707=26, srv1486190803=25, srv306807008=64, srv1142072580=5, srv1772912748=36, srv1073671122=1, srv913223977=94, srv1794875250=39, srv1952714031=48, srv88351242=90, srv1209738172=9, srv345735342=65, srv938423023=96, srv992951381=99, srv361420310=67, srv559839211=80, srv371875736=70, srv553451783=79, srv1357250365=20, srv1732616078=32, srv1200642332=8, srv198874522=51, srv896129139=91, srv1684827893=28, srv1331924925=19, srv1472158368=24, srv170121053=29, srv3616107=68, srv1427095674=22, srv1249084800=13, srv1793706794=37, srv1225149006=10, srv123675539=12, srv2114755469=59, srv725906170=86, srv1922493806=46, srv1331558790=18, srv2105350864=58, srv1793878669=38, srv1888127565=44, srv787228314=88, srv98152467=98, srv1446456836=23, srv524157225=77, srv423197526=72, srv1310248395=16, srv1842205637=42, srv903132673=92, srv1230618723=11, srv1957939575=49, srv2055210181=56, srv435743224=73, srv118004462=6, srv173468065=33, srv757363856=87, srv1772841098=35, srv1619807643=27, srv2020447734=55, srv1917173776=45, srv1112946066=4, srv1976113794=50, srv1076165402=2, srv1999568051=52, srv2015270801=54, srv370340639=69, srv1808080303=40, srv656757113=83, srv417316523=71, srv305991306=63, srv1865212614=43, srv268863680=62, srv1810559156=41, srv929107684=95, srv442789462=74, srv1421582046=21, srv200479936=53, srv1052288810=0, srv194963036=47, srv910161676=93, srv688522531=85, srv1261538529=14, srv1112568363=3, srv52637337=78, srv1770990673=34, srv1312488177=17, srv1194052408=7, srv1714537929=30, srv1716466189=31, srv465445978=75, srv237768655=60, srv2093848918=57, srv670774647=84, srv806734757=89, srv252292607=61, srv969359493=97, srv1279482853=15, srv518482735=76, srv600203098=81, srv649988250=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-23 16:53:13,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-23 16:53:13,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-23 16:53:13,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-23 16:53:13,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-23 16:53:13,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-23 16:53:13,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-23 16:53:13,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-23 16:53:13,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-23 16:53:13,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-23 16:53:13,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-23 16:53:13,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-23 16:53:13,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-23 16:53:13,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-23 16:53:13,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-23 16:53:13,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-23 16:53:13,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-23 16:53:13,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-23 16:53:13,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-23 16:53:13,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-23 16:53:13,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-23 16:53:13,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-23 16:53:13,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-23 16:53:13,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-23 16:53:13,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-23 16:53:13,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-23 16:53:13,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-23 16:53:13,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-23 16:53:13,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-23 16:53:13,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-23 16:53:13,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-23 16:53:13,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-23 16:53:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-23 16:53:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-23 16:53:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-23 16:53:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-23 16:53:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-23 16:53:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-23 16:53:13,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-23 16:53:13,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-23 16:53:13,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-23 16:53:13,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-23 16:53:13,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-23 16:53:13,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-23 16:53:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-23 16:53:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-23 16:53:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-23 16:53:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-23 16:53:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-23 16:53:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-23 16:53:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-23 16:53:13,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-23 16:53:13,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-23 16:53:13,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-23 16:53:13,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-23 16:53:13,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-23 16:53:13,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-23 16:53:13,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-23 16:53:13,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-23 16:53:13,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-23 16:53:13,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-23 16:53:13,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-23 16:53:13,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-23 16:53:13,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-23 16:53:13,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-23 16:53:13,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-23 16:53:13,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-23 16:53:13,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-23 16:53:13,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-23 16:53:13,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-23 16:53:13,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-23 16:53:13,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-23 16:53:13,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-23 16:53:13,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-23 16:53:13,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-23 16:53:13,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-23 16:53:13,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-23 16:53:13,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-23 16:53:13,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-23 16:53:13,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-23 16:53:13,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-23 16:53:13,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-23 16:53:13,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-23 16:53:13,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-23 16:53:13,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-23 16:53:13,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-23 16:53:13,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-23 16:53:13,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-23 16:53:13,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-23 16:53:13,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-23 16:53:13,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-23 16:53:13,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-23 16:53:13,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-23 16:53:13,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-23 16:53:13,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-23 16:53:13,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-23 16:53:13,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-23 16:53:13,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-23 16:53:13,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-23 16:53:13,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-23 16:53:13,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-23 16:53:13,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-23 16:53:13,335 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-23 16:53:13,336 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-23 16:53:13,336 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-23 16:53:13,755 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 654 ms to try 5000 different iterations. Found a solution that moves 4633 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18817154928229726. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03948445671381695, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.058607101686789634, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5147777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03948445671381695, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18391666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24558333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-23 16:53:13,784 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2093848918:80 , srv123675539:81 , srv1052288810:82 , srv1112946066:82 , srv1793706794:82 , srv903132673:82 , srv1073671122:83 , srv1684827893:83 , srv1917173776:83 , srv305991306:83 , srv969359493:83 , srv1472158368:84 , srv1486190803:84 , srv118004462:85 , srv1209738172:85 , srv1261538529:85 , srv1421582046:85 , srv1714537929:85 , srv1772912748:85 , srv1810559156:85 , srv1865212614:85 , srv1999568051:85 , srv2105350864:85 , srv252292607:85 , srv306807008:85 , srv3616107:85 , srv370340639:85 , srv417316523:85 , srv435743224:85 , srv465445978:85 , srv600203098:85 , srv649988250:85 , srv656757113:85 , srv787228314:85 , srv1076165402:86 , srv1194052408:86 , srv1249084800:86 , srv1279482853:86 , srv1331558790:86 , srv1331924925:86 , srv1357250365:86 , srv1574160707:86 , srv1772841098:86 , srv194963036:86 , srv198874522:86 , srv200479936:86 , srv2015270801:86 , srv518482735:86 , srv524157225:86 , srv98152467:86 , srv992951381:86 , srv1112568363:87 , srv1225149006:87 , srv1310248395:87 , srv1312488177:87 , srv1446456836:87 , srv1716466189:87 , srv1732616078:87 , srv1888127565:87 , srv1952714031:87 , srv1976113794:87 , srv237768655:87 , srv345735342:87 , srv353177431:87 , srv371875736:87 , srv423197526:87 , srv670774647:87 , srv806734757:87 , srv910161676:87 , srv938423023:87 , srv1142072580:88 , srv1230618723:88 , srv1619807643:88 , srv173468065:88 , srv1770990673:88 , srv1842205637:88 , srv361420310:88 , srv52637337:88 , srv559839211:88 , srv725906170:88 , srv757363856:88 , srv88351242:88 , srv896129139:88 , srv929107684:88 , srv1200642332:89 , srv1793878669:89 , srv1808080303:89 , srv1957939575:89 , srv2114755469:89 , srv688522531:89 , srv913223977:89 , srv1427095674:90 , srv170121053:90 , srv553451783:90 , srv2020447734:91 , srv2055210181:92 , srv1794875250:93 , srv1922493806:93 , srv442789462:93 , srv268863680:441 } 2023-07-23 16:53:13,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-23 16:53:13,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv353177431=66, srv1574160707=26, srv1486190803=25, srv306807008=64, srv1142072580=5, srv1772912748=36, srv1073671122=1, srv913223977=94, srv1794875250=39, srv1952714031=48, srv88351242=90, srv1209738172=9, srv345735342=65, srv938423023=96, srv992951381=99, srv361420310=67, srv559839211=80, srv371875736=70, srv553451783=79, srv1357250365=20, srv1732616078=32, srv1200642332=8, srv198874522=51, srv896129139=91, srv1684827893=28, srv1331924925=19, srv1472158368=24, srv170121053=29, srv3616107=68, srv1427095674=22, srv1249084800=13, srv1793706794=37, srv1225149006=10, srv123675539=12, srv2114755469=59, srv725906170=86, srv1922493806=46, srv1331558790=18, srv2105350864=58, srv1793878669=38, srv1888127565=44, srv787228314=88, srv98152467=98, srv1446456836=23, srv524157225=77, srv423197526=72, srv1310248395=16, srv1842205637=42, srv903132673=92, srv1230618723=11, srv1957939575=49, srv2055210181=56, srv435743224=73, srv118004462=6, srv173468065=33, srv757363856=87, srv1772841098=35, srv1619807643=27, srv2020447734=55, srv1917173776=45, srv1112946066=4, srv1976113794=50, srv1076165402=2, srv1999568051=52, srv2015270801=54, srv370340639=69, srv1808080303=40, srv656757113=83, srv417316523=71, srv305991306=63, srv1865212614=43, srv268863680=62, srv1810559156=41, srv929107684=95, srv442789462=74, srv1421582046=21, srv200479936=53, srv1052288810=0, srv194963036=47, srv910161676=93, srv688522531=85, srv1261538529=14, srv1112568363=3, srv52637337=78, srv1770990673=34, srv1312488177=17, srv1194052408=7, srv1714537929=30, srv1716466189=31, srv465445978=75, srv237768655=60, srv2093848918=57, srv670774647=84, srv806734757=89, srv252292607=61, srv969359493=97, srv1279482853=15, srv518482735=76, srv600203098=81, srv649988250=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-23 16:53:13,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-23 16:53:13,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-23 16:53:13,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-23 16:53:13,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-23 16:53:13,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-23 16:53:13,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-23 16:53:13,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-23 16:53:13,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-23 16:53:13,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-23 16:53:13,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-23 16:53:13,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-23 16:53:13,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-23 16:53:13,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-23 16:53:13,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-23 16:53:13,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-23 16:53:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-23 16:53:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-23 16:53:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-23 16:53:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-23 16:53:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-23 16:53:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-23 16:53:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-23 16:53:13,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-23 16:53:13,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-23 16:53:13,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-23 16:53:13,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-23 16:53:13,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-23 16:53:13,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-23 16:53:13,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-23 16:53:13,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-23 16:53:13,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-23 16:53:13,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-23 16:53:13,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-23 16:53:13,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-23 16:53:13,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-23 16:53:13,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-23 16:53:13,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-23 16:53:13,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-23 16:53:13,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-23 16:53:13,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-23 16:53:13,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-23 16:53:13,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-23 16:53:13,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-23 16:53:13,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-23 16:53:13,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-23 16:53:13,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-23 16:53:13,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-23 16:53:13,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-23 16:53:13,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-23 16:53:13,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-23 16:53:13,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-23 16:53:13,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-23 16:53:13,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-23 16:53:13,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-23 16:53:13,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-23 16:53:13,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-23 16:53:13,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-23 16:53:13,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-23 16:53:13,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-23 16:53:13,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-23 16:53:13,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-23 16:53:13,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-23 16:53:13,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-23 16:53:13,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-23 16:53:13,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-23 16:53:13,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-23 16:53:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-23 16:53:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-23 16:53:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-23 16:53:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-23 16:53:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-23 16:53:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-23 16:53:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-23 16:53:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-23 16:53:13,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-23 16:53:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-23 16:53:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-23 16:53:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-23 16:53:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-23 16:53:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-23 16:53:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-23 16:53:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-23 16:53:13,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-23 16:53:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-23 16:53:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-23 16:53:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-23 16:53:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-23 16:53:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-23 16:53:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-23 16:53:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-23 16:53:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-23 16:53:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-23 16:53:13,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-23 16:53:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-23 16:53:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-23 16:53:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-23 16:53:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-23 16:53:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-23 16:53:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-23 16:53:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-23 16:53:13,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-23 16:53:14,005 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03948445671381695, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.058607101686789634, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03948445671381695, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18391666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24558333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-23 16:53:14,005 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-23 16:53:14,006 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18813910832292272, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03948445671381695, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.058607101686789634, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03948445671381695, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18391666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24558333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-23 16:53:14,401 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 540 ms to try 5000 different iterations. Found a solution that moves 2641 regions; Going from a computed imbalance of 0.18813910832292272 to a new imbalance of 0.01675474229125175. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019929417790743153); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03336698637909104, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29344444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019929417790743153); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04733333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-23 16:53:14,414 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1794875250:77 , srv2020447734:80 , srv442789462:80 , srv1793878669:81 , srv1922493806:81 , srv1957939575:81 , srv553451783:81 , srv1427095674:82 , srv2055210181:82 , srv2114755469:82 , srv1230618723:83 , srv52637337:83 , srv725906170:83 , srv757363856:83 , srv88351242:83 , srv1142072580:84 , srv1194052408:84 , srv1225149006:84 , srv170121053:84 , srv173468065:84 , srv1808080303:84 , srv1842205637:84 , srv1952714031:84 , srv1976113794:84 , srv198874522:84 , srv896129139:84 , srv913223977:84 , srv938423023:84 , srv1076165402:85 , srv118004462:85 , srv1200642332:85 , srv1261538529:85 , srv1331558790:85 , srv1446456836:85 , srv1574160707:85 , srv1619807643:85 , srv1732616078:85 , srv1770990673:85 , srv1772841098:85 , srv1772912748:85 , srv237768655:85 , srv306807008:85 , srv345735342:85 , srv361420310:85 , srv688522531:85 , srv1112568363:86 , srv1249084800:86 , srv1486190803:86 , srv1888127565:86 , srv194963036:86 , srv2015270801:86 , srv370340639:86 , srv417316523:86 , srv423197526:86 , srv524157225:86 , srv559839211:86 , srv649988250:86 , srv787228314:86 , srv910161676:86 , srv929107684:86 , srv1209738172:87 , srv1279482853:87 , srv1310248395:87 , srv1312488177:87 , srv1331924925:87 , srv1421582046:87 , srv1472158368:87 , srv1810559156:87 , srv1999568051:87 , srv200479936:87 , srv2105350864:87 , srv353177431:87 , srv371875736:87 , srv435743224:87 , srv465445978:87 , srv656757113:87 , srv1357250365:88 , srv1714537929:88 , srv1865212614:88 , srv252292607:88 , srv518482735:88 , srv600203098:88 , srv670774647:88 , srv903132673:88 , srv969359493:88 , srv98152467:88 , srv992951381:88 , srv1073671122:89 , srv1112946066:89 , srv1716466189:89 , srv1793706794:89 , srv3616107:89 , srv806734757:89 , srv1052288810:90 , srv305991306:90 , srv123675539:91 , srv1684827893:92 , srv1917173776:92 , srv2093848918:92 , srv268863680:510 } 2023-07-23 16:53:14,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-23 16:53:14,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv353177431=66, srv1574160707=26, srv1486190803=25, srv306807008=64, srv1142072580=5, srv1772912748=36, srv1073671122=1, srv913223977=94, srv1794875250=39, srv1952714031=48, srv88351242=90, srv1209738172=9, srv345735342=65, srv938423023=96, srv992951381=99, srv361420310=67, srv559839211=80, srv371875736=70, srv553451783=79, srv1357250365=20, srv1732616078=32, srv1200642332=8, srv198874522=51, srv896129139=91, srv1684827893=28, srv1331924925=19, srv1472158368=24, srv170121053=29, srv3616107=68, srv1427095674=22, srv1249084800=13, srv1793706794=37, srv1225149006=10, srv123675539=12, srv2114755469=59, srv725906170=86, srv1922493806=46, srv1331558790=18, srv2105350864=58, srv1793878669=38, srv1888127565=44, srv787228314=88, srv98152467=98, srv1446456836=23, srv524157225=77, srv423197526=72, srv1310248395=16, srv1842205637=42, srv903132673=92, srv1230618723=11, srv1957939575=49, srv2055210181=56, srv435743224=73, srv118004462=6, srv173468065=33, srv757363856=87, srv1772841098=35, srv1619807643=27, srv2020447734=55, srv1917173776=45, srv1112946066=4, srv1976113794=50, srv1076165402=2, srv1999568051=52, srv2015270801=54, srv370340639=69, srv1808080303=40, srv656757113=83, srv417316523=71, srv305991306=63, srv1865212614=43, srv268863680=62, srv1810559156=41, srv929107684=95, srv442789462=74, srv1421582046=21, srv200479936=53, srv1052288810=0, srv194963036=47, srv910161676=93, srv688522531=85, srv1261538529=14, srv1112568363=3, srv52637337=78, srv1770990673=34, srv1312488177=17, srv1194052408=7, srv1714537929=30, srv1716466189=31, srv465445978=75, srv237768655=60, srv2093848918=57, srv670774647=84, srv806734757=89, srv252292607=61, srv969359493=97, srv1279482853=15, srv518482735=76, srv600203098=81, srv649988250=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-23 16:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-23 16:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-23 16:53:14,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-23 16:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-23 16:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-23 16:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-23 16:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-23 16:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-23 16:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-23 16:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-23 16:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-23 16:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-23 16:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-23 16:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-23 16:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-23 16:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-23 16:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-23 16:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-23 16:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-23 16:53:14,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-23 16:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-23 16:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-23 16:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-23 16:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-23 16:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-23 16:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-23 16:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-23 16:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-23 16:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-23 16:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-23 16:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-23 16:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-23 16:53:14,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-23 16:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-23 16:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-23 16:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-23 16:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-23 16:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-23 16:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-23 16:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-23 16:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-23 16:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-23 16:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-23 16:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-23 16:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-23 16:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-23 16:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-23 16:53:14,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-23 16:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-23 16:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-23 16:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-23 16:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-23 16:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-23 16:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-23 16:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-23 16:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-23 16:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-23 16:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-23 16:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-23 16:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-23 16:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-23 16:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-23 16:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-23 16:53:14,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-23 16:53:14,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-23 16:53:14,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-23 16:53:14,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-23 16:53:14,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-23 16:53:14,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-23 16:53:14,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-23 16:53:14,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-23 16:53:14,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-23 16:53:14,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-23 16:53:14,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-23 16:53:14,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-23 16:53:14,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-23 16:53:14,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-23 16:53:14,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-23 16:53:14,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-23 16:53:14,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-23 16:53:14,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-23 16:53:14,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-23 16:53:14,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-23 16:53:14,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-23 16:53:14,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-23 16:53:14,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-23 16:53:14,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-23 16:53:14,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-23 16:53:14,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-23 16:53:14,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-23 16:53:14,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-23 16:53:14,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-23 16:53:14,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-23 16:53:14,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-23 16:53:14,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-23 16:53:14,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-23 16:53:14,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-23 16:53:14,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-23 16:53:14,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-23 16:53:14,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-23 16:53:14,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-23 16:53:14,577 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019929417790743153); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03336698637909104, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019929417790743153); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04733333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-23 16:53:14,577 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-23 16:53:14,577 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016736249613999835, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019929417790743153); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03336698637909104, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019929417790743153); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04733333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-23 16:53:14,918 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 463 ms to try 5000 different iterations. Found a solution that moves 1031 regions; Going from a computed imbalance of 0.016736249613999835 to a new imbalance of 5.845219109452483E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005757280398827387); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010423146132940955); MoveCostFunction : (multiplier=7.0, imbalance=0.11455555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005757280398827387); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-23 16:53:14,921 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv806734757:81 , srv1716466189:82 , srv1917173776:82 , srv1684827893:83 , srv170121053:83 , srv2055210181:83 , srv353177431:83 , srv3616107:83 , srv371875736:83 , srv559839211:83 , srv670774647:83 , srv98152467:83 , srv1200642332:84 , srv1310248395:84 , srv1312488177:84 , srv1357250365:84 , srv1770990673:84 , srv1888127565:84 , srv1922493806:84 , srv200479936:84 , srv252292607:84 , srv305991306:84 , srv423197526:84 , srv442789462:84 , srv518482735:84 , srv688522531:84 , srv910161676:84 , srv913223977:84 , srv929107684:84 , srv992951381:84 , srv1073671122:85 , srv1112568363:85 , srv1209738172:85 , srv123675539:85 , srv1279482853:85 , srv1331924925:85 , srv1427095674:85 , srv1446456836:85 , srv1619807643:85 , srv1714537929:85 , srv1732616078:85 , srv173468065:85 , srv1808080303:85 , srv1810559156:85 , srv1842205637:85 , srv1865212614:85 , srv1999568051:85 , srv2015270801:85 , srv237768655:85 , srv345735342:85 , srv361420310:85 , srv600203098:85 , srv1052288810:86 , srv1076165402:86 , srv1112946066:86 , srv1142072580:86 , srv1225149006:86 , srv1230618723:86 , srv1249084800:86 , srv1331558790:86 , srv1421582046:86 , srv1574160707:86 , srv1793706794:86 , srv1793878669:86 , srv194963036:86 , srv1952714031:86 , srv2093848918:86 , srv2105350864:86 , srv370340639:86 , srv435743224:86 , srv524157225:86 , srv52637337:86 , srv649988250:86 , srv656757113:86 , srv757363856:86 , srv787228314:86 , srv896129139:86 , srv969359493:86 , srv118004462:87 , srv1194052408:87 , srv1472158368:87 , srv1486190803:87 , srv1772841098:87 , srv1794875250:87 , srv1976113794:87 , srv2020447734:87 , srv2114755469:87 , srv306807008:87 , srv417316523:87 , srv465445978:87 , srv553451783:87 , srv725906170:87 , srv88351242:87 , srv903132673:87 , srv1261538529:88 , srv1772912748:88 , srv1957939575:88 , srv198874522:88 , srv938423023:88 , srv268863680:558 } 2023-07-23 16:53:14,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-23 16:53:14,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv353177431=66, srv1574160707=26, srv1486190803=25, srv306807008=64, srv1142072580=5, srv1772912748=36, srv1073671122=1, srv913223977=94, srv1794875250=39, srv1952714031=48, srv88351242=90, srv1209738172=9, srv345735342=65, srv938423023=96, srv992951381=99, srv361420310=67, srv559839211=80, srv371875736=70, srv553451783=79, srv1357250365=20, srv1732616078=32, srv1200642332=8, srv198874522=51, srv896129139=91, srv1684827893=28, srv1331924925=19, srv1472158368=24, srv170121053=29, srv3616107=68, srv1427095674=22, srv1249084800=13, srv1793706794=37, srv1225149006=10, srv123675539=12, srv2114755469=59, srv725906170=86, srv1922493806=46, srv1331558790=18, srv2105350864=58, srv1793878669=38, srv1888127565=44, srv787228314=88, srv98152467=98, srv1446456836=23, srv524157225=77, srv423197526=72, srv1310248395=16, srv1842205637=42, srv903132673=92, srv1230618723=11, srv1957939575=49, srv2055210181=56, srv435743224=73, srv118004462=6, srv173468065=33, srv757363856=87, srv1772841098=35, srv1619807643=27, srv2020447734=55, srv1917173776=45, srv1112946066=4, srv1976113794=50, srv1076165402=2, srv1999568051=52, srv2015270801=54, srv370340639=69, srv1808080303=40, srv656757113=83, srv417316523=71, srv305991306=63, srv1865212614=43, srv268863680=62, srv1810559156=41, srv929107684=95, srv442789462=74, srv1421582046=21, srv200479936=53, srv1052288810=0, srv194963036=47, srv910161676=93, srv688522531=85, srv1261538529=14, srv1112568363=3, srv52637337=78, srv1770990673=34, srv1312488177=17, srv1194052408=7, srv1714537929=30, srv1716466189=31, srv465445978=75, srv237768655=60, srv2093848918=57, srv670774647=84, srv806734757=89, srv252292607=61, srv969359493=97, srv1279482853=15, srv518482735=76, srv600203098=81, srv649988250=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-23 16:53:14,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-23 16:53:14,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-23 16:53:14,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-23 16:53:14,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-23 16:53:14,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-23 16:53:14,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-23 16:53:14,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-23 16:53:14,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-23 16:53:14,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-23 16:53:14,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-23 16:53:14,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-23 16:53:14,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-23 16:53:14,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-23 16:53:14,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-23 16:53:14,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-23 16:53:14,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-23 16:53:14,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-23 16:53:14,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-23 16:53:14,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-23 16:53:14,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-23 16:53:14,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-23 16:53:14,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-23 16:53:14,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-23 16:53:14,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-23 16:53:14,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-23 16:53:14,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-23 16:53:14,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-23 16:53:14,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-23 16:53:14,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-23 16:53:14,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-23 16:53:14,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-23 16:53:14,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-23 16:53:14,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-23 16:53:14,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-23 16:53:14,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-23 16:53:14,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-23 16:53:14,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-23 16:53:14,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-23 16:53:14,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-23 16:53:14,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-23 16:53:14,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-23 16:53:14,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-23 16:53:14,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-23 16:53:14,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-23 16:53:14,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-23 16:53:14,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-23 16:53:14,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-23 16:53:14,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-23 16:53:14,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-23 16:53:14,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-23 16:53:14,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-23 16:53:14,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-23 16:53:14,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-23 16:53:14,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-23 16:53:14,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-23 16:53:14,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-23 16:53:14,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-23 16:53:14,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-23 16:53:14,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-23 16:53:14,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-23 16:53:14,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-23 16:53:14,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-23 16:53:14,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-23 16:53:14,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-23 16:53:14,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-23 16:53:14,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-23 16:53:14,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-23 16:53:14,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-23 16:53:14,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-23 16:53:14,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-23 16:53:14,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-23 16:53:14,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-23 16:53:14,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-23 16:53:14,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-23 16:53:14,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-23 16:53:14,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-23 16:53:14,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-23 16:53:14,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-23 16:53:14,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-23 16:53:14,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-23 16:53:14,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-23 16:53:14,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-23 16:53:14,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-23 16:53:14,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-23 16:53:14,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-23 16:53:14,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-23 16:53:14,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-23 16:53:14,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-23 16:53:14,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-23 16:53:14,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-23 16:53:14,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-23 16:53:14,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-23 16:53:14,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-23 16:53:14,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-23 16:53:14,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-23 16:53:14,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-23 16:53:14,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-23 16:53:14,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-23 16:53:14,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-23 16:53:14,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-23 16:53:14,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-23 16:53:15,125 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005757280398827387); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010423146132940955); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005757280398827387); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-23 16:53:15,125 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-23 16:53:15,125 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.773026946458444E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005757280398827387); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010423146132940955); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005757280398827387); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-23 16:53:15,498 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 513 ms to try 5000 different iterations. Found a solution that moves 252 regions; Going from a computed imbalance of 5.773026946458444E-4 to a new imbalance of 2.855472184230623E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.028, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-23 16:53:15,500 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1714537929:83 , srv465445978:83 , srv725906170:83 , srv88351242:83 , srv938423023:83 , srv1052288810:84 , srv1142072580:84 , srv1200642332:84 , srv1225149006:84 , srv1249084800:84 , srv1261538529:84 , srv1279482853:84 , srv1310248395:84 , srv1312488177:84 , srv1331924925:84 , srv1357250365:84 , srv1421582046:84 , srv1472158368:84 , srv1619807643:84 , srv1684827893:84 , srv1716466189:84 , srv1772841098:84 , srv1772912748:84 , srv1808080303:84 , srv1865212614:84 , srv194963036:84 , srv1957939575:84 , srv1976113794:84 , srv2020447734:84 , srv2093848918:84 , srv2105350864:84 , srv2114755469:84 , srv361420310:84 , srv371875736:84 , srv417316523:84 , srv435743224:84 , srv518482735:84 , srv524157225:84 , srv600203098:84 , srv649988250:84 , srv670774647:84 , srv688522531:84 , srv896129139:84 , srv929107684:84 , srv992951381:84 , srv1073671122:85 , srv1076165402:85 , srv1112568363:85 , srv1112946066:85 , srv118004462:85 , srv1194052408:85 , srv1209738172:85 , srv123675539:85 , srv1331558790:85 , srv1427095674:85 , srv1486190803:85 , srv1574160707:85 , srv170121053:85 , srv1732616078:85 , srv173468065:85 , srv1770990673:85 , srv1793706794:85 , srv1793878669:85 , srv1794875250:85 , srv1810559156:85 , srv1842205637:85 , srv1888127565:85 , srv1917173776:85 , srv1922493806:85 , srv1952714031:85 , srv198874522:85 , srv1999568051:85 , srv200479936:85 , srv2015270801:85 , srv2055210181:85 , srv237768655:85 , srv252292607:85 , srv305991306:85 , srv306807008:85 , srv345735342:85 , srv353177431:85 , srv3616107:85 , srv370340639:85 , srv423197526:85 , srv442789462:85 , srv52637337:85 , srv553451783:85 , srv559839211:85 , srv656757113:85 , srv757363856:85 , srv787228314:85 , srv806734757:85 , srv903132673:85 , srv910161676:85 , srv913223977:85 , srv969359493:85 , srv98152467:85 , srv1230618723:86 , srv1446456836:86 , srv268863680:633 } 2023-07-23 16:53:15,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-23 16:53:15,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv353177431=66, srv1574160707=26, srv1486190803=25, srv306807008=64, srv1142072580=5, srv1772912748=36, srv1073671122=1, srv913223977=94, srv1794875250=39, srv1952714031=48, srv88351242=90, srv1209738172=9, srv345735342=65, srv938423023=96, srv992951381=99, srv361420310=67, srv559839211=80, srv371875736=70, srv553451783=79, srv1357250365=20, srv1732616078=32, srv1200642332=8, srv198874522=51, srv896129139=91, srv1684827893=28, srv1331924925=19, srv1472158368=24, srv170121053=29, srv3616107=68, srv1427095674=22, srv1249084800=13, srv1793706794=37, srv1225149006=10, srv123675539=12, srv2114755469=59, srv725906170=86, srv1922493806=46, srv1331558790=18, srv2105350864=58, srv1793878669=38, srv1888127565=44, srv787228314=88, srv98152467=98, srv1446456836=23, srv524157225=77, srv423197526=72, srv1310248395=16, srv1842205637=42, srv903132673=92, srv1230618723=11, srv1957939575=49, srv2055210181=56, srv435743224=73, srv118004462=6, srv173468065=33, srv757363856=87, srv1772841098=35, srv1619807643=27, srv2020447734=55, srv1917173776=45, srv1112946066=4, srv1976113794=50, srv1076165402=2, srv1999568051=52, srv2015270801=54, srv370340639=69, srv1808080303=40, srv656757113=83, srv417316523=71, srv305991306=63, srv1865212614=43, srv268863680=62, srv1810559156=41, srv929107684=95, srv442789462=74, srv1421582046=21, srv200479936=53, srv1052288810=0, srv194963036=47, srv910161676=93, srv688522531=85, srv1261538529=14, srv1112568363=3, srv52637337=78, srv1770990673=34, srv1312488177=17, srv1194052408=7, srv1714537929=30, srv1716466189=31, srv465445978=75, srv237768655=60, srv2093848918=57, srv670774647=84, srv806734757=89, srv252292607=61, srv969359493=97, srv1279482853=15, srv518482735=76, srv600203098=81, srv649988250=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-23 16:53:15,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-23 16:53:15,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-23 16:53:15,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-23 16:53:15,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-23 16:53:15,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-23 16:53:15,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-23 16:53:15,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-23 16:53:15,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-23 16:53:15,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-23 16:53:15,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-23 16:53:15,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-23 16:53:15,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-23 16:53:15,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-23 16:53:15,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-23 16:53:15,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-23 16:53:15,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-23 16:53:15,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-23 16:53:15,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-23 16:53:15,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-23 16:53:15,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-23 16:53:15,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-23 16:53:15,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-23 16:53:15,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-23 16:53:15,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-23 16:53:15,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-23 16:53:15,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-23 16:53:15,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-23 16:53:15,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-23 16:53:15,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-23 16:53:15,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-23 16:53:15,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-23 16:53:15,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-23 16:53:15,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-23 16:53:15,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-23 16:53:15,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-23 16:53:15,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-23 16:53:15,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-23 16:53:15,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-23 16:53:15,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-23 16:53:15,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-23 16:53:15,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-23 16:53:15,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-23 16:53:15,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-23 16:53:15,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-23 16:53:15,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-23 16:53:15,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-23 16:53:15,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-23 16:53:15,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-23 16:53:15,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-23 16:53:15,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-23 16:53:15,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-23 16:53:15,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-23 16:53:15,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-23 16:53:15,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-23 16:53:15,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-23 16:53:15,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-23 16:53:15,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-23 16:53:15,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-23 16:53:15,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-23 16:53:15,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-23 16:53:15,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-23 16:53:15,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-23 16:53:15,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-23 16:53:15,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-23 16:53:15,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-23 16:53:15,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-23 16:53:15,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-23 16:53:15,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-23 16:53:15,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-23 16:53:15,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-23 16:53:15,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-23 16:53:15,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-23 16:53:15,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-23 16:53:15,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-23 16:53:15,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-23 16:53:15,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-23 16:53:15,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-23 16:53:15,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-23 16:53:15,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-23 16:53:15,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-23 16:53:15,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-23 16:53:15,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-23 16:53:15,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-23 16:53:15,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-23 16:53:15,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-23 16:53:15,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-23 16:53:15,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-23 16:53:15,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-23 16:53:15,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-23 16:53:15,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-23 16:53:15,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-23 16:53:15,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-23 16:53:15,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-23 16:53:15,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-23 16:53:15,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-23 16:53:15,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-23 16:53:15,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-23 16:53:15,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-23 16:53:15,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-23 16:53:15,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-23 16:53:15,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-23 16:53:15,658 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-23 16:53:15,659 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-23 16:53:15,659 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.6790180127999938E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0021711298242630987); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-23 16:53:15,972 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 427 ms to try 5000 different iterations. Found a solution that moves 23 regions; Going from a computed imbalance of 2.6790180127999938E-5 to a new imbalance of 8.420813428170152E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0025555555555555557); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-23 16:53:15,972 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1112568363:83 , srv1230618723:83 , srv1446456836:83 , srv1922493806:83 , srv198874522:83 , srv553451783:83 , srv656757113:83 , srv1052288810:84 , srv1073671122:84 , srv1076165402:84 , srv1112946066:84 , srv1142072580:84 , srv118004462:84 , srv1194052408:84 , srv1200642332:84 , srv1209738172:84 , srv123675539:84 , srv1249084800:84 , srv1261538529:84 , srv1279482853:84 , srv1310248395:84 , srv1312488177:84 , srv1331558790:84 , srv1331924925:84 , srv1357250365:84 , srv1421582046:84 , srv1427095674:84 , srv1472158368:84 , srv1486190803:84 , srv1574160707:84 , srv1619807643:84 , srv1684827893:84 , srv170121053:84 , srv1716466189:84 , srv1732616078:84 , srv173468065:84 , srv1770990673:84 , srv1772841098:84 , srv1772912748:84 , srv1793706794:84 , srv1794875250:84 , srv1808080303:84 , srv1810559156:84 , srv1842205637:84 , srv1865212614:84 , srv1888127565:84 , srv1917173776:84 , srv194963036:84 , srv1952714031:84 , srv1957939575:84 , srv1976113794:84 , srv1999568051:84 , srv200479936:84 , srv2015270801:84 , srv2020447734:84 , srv2055210181:84 , srv2093848918:84 , srv2105350864:84 , srv2114755469:84 , srv237768655:84 , srv252292607:84 , srv305991306:84 , srv306807008:84 , srv345735342:84 , srv353177431:84 , srv361420310:84 , srv3616107:84 , srv370340639:84 , srv417316523:84 , srv423197526:84 , srv435743224:84 , srv442789462:84 , srv465445978:84 , srv518482735:84 , srv524157225:84 , srv52637337:84 , srv559839211:84 , srv600203098:84 , srv670774647:84 , srv688522531:84 , srv757363856:84 , srv787228314:84 , srv806734757:84 , srv896129139:84 , srv903132673:84 , srv910161676:84 , srv913223977:84 , srv929107684:84 , srv938423023:84 , srv969359493:84 , srv98152467:84 , srv992951381:84 , srv1225149006:85 , srv1714537929:85 , srv1793878669:85 , srv371875736:85 , srv649988250:85 , srv725906170:85 , srv88351242:85 , srv268863680:684 } 2023-07-23 16:53:15,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-23 16:53:15,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv353177431=66, srv1574160707=26, srv1486190803=25, srv306807008=64, srv1142072580=5, srv1772912748=36, srv1073671122=1, srv913223977=94, srv1794875250=39, srv1952714031=48, srv88351242=90, srv1209738172=9, srv345735342=65, srv938423023=96, srv992951381=99, srv361420310=67, srv559839211=80, srv371875736=70, srv553451783=79, srv1357250365=20, srv1732616078=32, srv1200642332=8, srv198874522=51, srv896129139=91, srv1684827893=28, srv1331924925=19, srv1472158368=24, srv170121053=29, srv3616107=68, srv1427095674=22, srv1249084800=13, srv1793706794=37, srv1225149006=10, srv123675539=12, srv2114755469=59, srv725906170=86, srv1922493806=46, srv1331558790=18, srv2105350864=58, srv1793878669=38, srv1888127565=44, srv787228314=88, srv98152467=98, srv1446456836=23, srv524157225=77, srv423197526=72, srv1310248395=16, srv1842205637=42, srv903132673=92, srv1230618723=11, srv1957939575=49, srv2055210181=56, srv435743224=73, srv118004462=6, srv173468065=33, srv757363856=87, srv1772841098=35, srv1619807643=27, srv2020447734=55, srv1917173776=45, srv1112946066=4, srv1976113794=50, srv1076165402=2, srv1999568051=52, srv2015270801=54, srv370340639=69, srv1808080303=40, srv656757113=83, srv417316523=71, srv305991306=63, srv1865212614=43, srv268863680=62, srv1810559156=41, srv929107684=95, srv442789462=74, srv1421582046=21, srv200479936=53, srv1052288810=0, srv194963036=47, srv910161676=93, srv688522531=85, srv1261538529=14, srv1112568363=3, srv52637337=78, srv1770990673=34, srv1312488177=17, srv1194052408=7, srv1714537929=30, srv1716466189=31, srv465445978=75, srv237768655=60, srv2093848918=57, srv670774647=84, srv806734757=89, srv252292607=61, srv969359493=97, srv1279482853=15, srv518482735=76, srv600203098=81, srv649988250=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-23 16:53:15,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-23 16:53:15,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-23 16:53:15,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-23 16:53:15,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-23 16:53:15,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-23 16:53:15,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-23 16:53:15,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-23 16:53:15,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-23 16:53:15,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-23 16:53:15,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-23 16:53:15,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-23 16:53:15,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-23 16:53:15,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-23 16:53:15,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-23 16:53:15,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-23 16:53:15,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-23 16:53:15,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-23 16:53:15,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-23 16:53:15,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-23 16:53:15,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-23 16:53:15,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-23 16:53:15,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-23 16:53:15,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-23 16:53:15,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-23 16:53:15,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-23 16:53:15,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-23 16:53:15,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-23 16:53:15,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-23 16:53:15,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-23 16:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-23 16:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-23 16:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-23 16:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-23 16:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-23 16:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-23 16:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-23 16:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-23 16:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-23 16:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-23 16:53:15,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-23 16:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-23 16:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-23 16:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-23 16:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-23 16:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-23 16:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-23 16:53:15,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-23 16:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-23 16:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-23 16:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-23 16:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-23 16:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-23 16:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-23 16:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-23 16:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-23 16:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-23 16:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-23 16:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-23 16:53:15,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-23 16:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-23 16:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-23 16:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-23 16:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-23 16:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-23 16:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-23 16:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-23 16:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-23 16:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-23 16:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-23 16:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-23 16:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-23 16:53:15,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-23 16:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-23 16:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-23 16:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-23 16:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-23 16:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-23 16:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-23 16:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-23 16:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-23 16:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-23 16:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-23 16:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-23 16:53:16,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-23 16:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-23 16:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-23 16:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-23 16:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-23 16:53:16,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-23 16:53:16,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-23 16:53:16,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-23 16:53:16,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-23 16:53:16,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-23 16:53:16,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-23 16:53:16,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-23 16:53:16,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-23 16:53:16,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-23 16:53:16,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-23 16:53:16,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-23 16:53:16,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-23 16:53:16,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-23 16:53:16,163 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=8.259763985991403E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-23 16:53:16,164 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1112568363:83 , srv1230618723:83 , srv1446456836:83 , srv1922493806:83 , srv198874522:83 , srv553451783:83 , srv656757113:83 , srv1052288810:84 , srv1073671122:84 , srv1076165402:84 , srv1112946066:84 , srv1142072580:84 , srv118004462:84 , srv1194052408:84 , srv1200642332:84 , srv1209738172:84 , srv123675539:84 , srv1249084800:84 , srv1261538529:84 , srv1279482853:84 , srv1310248395:84 , srv1312488177:84 , srv1331558790:84 , srv1331924925:84 , srv1357250365:84 , srv1421582046:84 , srv1427095674:84 , srv1472158368:84 , srv1486190803:84 , srv1574160707:84 , srv1619807643:84 , srv1684827893:84 , srv170121053:84 , srv1716466189:84 , srv1732616078:84 , srv173468065:84 , srv1770990673:84 , srv1772841098:84 , srv1772912748:84 , srv1793706794:84 , srv1794875250:84 , srv1808080303:84 , srv1810559156:84 , srv1842205637:84 , srv1865212614:84 , srv1888127565:84 , srv1917173776:84 , srv194963036:84 , srv1952714031:84 , srv1957939575:84 , srv1976113794:84 , srv1999568051:84 , srv200479936:84 , srv2015270801:84 , srv2020447734:84 , srv2055210181:84 , srv2093848918:84 , srv2105350864:84 , srv2114755469:84 , srv237768655:84 , srv252292607:84 , srv305991306:84 , srv306807008:84 , srv345735342:84 , srv353177431:84 , srv361420310:84 , srv3616107:84 , srv370340639:84 , srv417316523:84 , srv423197526:84 , srv435743224:84 , srv442789462:84 , srv465445978:84 , srv518482735:84 , srv524157225:84 , srv52637337:84 , srv559839211:84 , srv600203098:84 , srv670774647:84 , srv688522531:84 , srv757363856:84 , srv787228314:84 , srv806734757:84 , srv896129139:84 , srv903132673:84 , srv910161676:84 , srv913223977:84 , srv929107684:84 , srv938423023:84 , srv969359493:84 , srv98152467:84 , srv992951381:84 , srv1225149006:85 , srv1714537929:85 , srv1793878669:85 , srv371875736:85 , srv649988250:85 , srv725906170:85 , srv88351242:85 , srv268863680:684 } 2023-07-23 16:53:16,198 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=294 (was 241) - SystemLoadAverage LEAK? -, ProcessCount=175 (was 175), AvailableMemoryMB=9800 (was 10561) 2023-07-23 16:53:16,209 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=294, ProcessCount=175, AvailableMemoryMB=9799 2023-07-23 16:53:16,210 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-23 16:53:16,211 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-07-23 16:53:16,223 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1020936385:84 , srv1043849478:84 , srv1097437830:84 , srv1108495378:84 , srv1129914153:84 , srv120059140:84 , srv1226843480:84 , srv1230480169:84 , srv1234077708:84 , srv1247075104:84 , srv1341675059:84 , srv1352107377:84 , srv1385407332:84 , srv1424862959:84 , srv1438340768:84 , srv1635464456:84 , srv1709941459:84 , srv1920066092:84 , srv2099547683:84 , srv266477452:84 , srv418394859:84 , srv44353901:84 , srv453583932:84 , srv512572293:84 , srv556879481:84 , srv576829177:84 , srv688755869:84 , srv72145975:84 , srv906482856:84 , srv1416325396:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-07-23 16:53:16,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-23 16:53:16,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1230480169=7, srv1129914153=4, srv44353901=22, srv1247075104=9, srv688755869=27, srv418394859=21, srv1709941459=17, srv72145975=28, srv266477452=20, srv1635464456=16, srv2099547683=19, srv1352107377=11, srv556879481=25, srv453583932=23, srv120059140=5, srv1234077708=8, srv1438340768=15, srv1043849478=1, srv576829177=26, srv1226843480=6, srv1341675059=10, srv1424862959=14, srv512572293=24, srv1097437830=2, srv1108495378=3, srv1385407332=12, srv1416325396=13, srv1920066092=18, srv906482856=29, srv1020936385=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-23 16:53:16,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-23 16:53:16,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-23 16:53:16,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-23 16:53:16,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-23 16:53:16,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-23 16:53:16,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-23 16:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-23 16:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-23 16:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-23 16:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-23 16:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-23 16:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-23 16:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-23 16:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-23 16:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-23 16:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-23 16:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-23 16:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-23 16:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-23 16:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-23 16:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-23 16:53:16,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-23 16:53:16,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-23 16:53:16,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-23 16:53:16,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-23 16:53:16,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-23 16:53:16,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-23 16:53:16,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-23 16:53:16,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-23 16:53:16,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-23 16:53:16,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-07-23 16:53:16,271 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1015214550647539, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-23 16:53:16,271 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912512934054208, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1015214550647539, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-07-23 16:55:16,249 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5585418 different iterations. Found a solution that moves 1851 regions; Going from a computed imbalance of 0.9912512934054208 to a new imbalance of 4.320326340186438E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6855555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-23 16:55:16,255 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1020936385:90 , srv1043849478:90 , srv1097437830:90 , srv1108495378:90 , srv1129914153:90 , srv120059140:90 , srv1226843480:90 , srv1230480169:90 , srv1234077708:90 , srv1247075104:90 , srv1341675059:90 , srv1352107377:90 , srv1385407332:90 , srv1416325396:90 , srv1424862959:90 , srv1438340768:90 , srv1635464456:90 , srv1709941459:90 , srv1920066092:90 , srv2099547683:90 , srv266477452:90 , srv418394859:90 , srv44353901:90 , srv453583932:90 , srv512572293:90 , srv556879481:90 , srv576829177:90 , srv688755869:90 , srv72145975:90 , srv906482856:90 } 2023-07-23 16:55:16,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-23 16:55:16,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1230480169=7, srv1129914153=4, srv44353901=22, srv1247075104=9, srv688755869=27, srv418394859=21, srv1709941459=17, srv72145975=28, srv266477452=20, srv1635464456=16, srv2099547683=19, srv1352107377=11, srv556879481=25, srv453583932=23, srv120059140=5, srv1234077708=8, srv1438340768=15, srv1043849478=1, srv576829177=26, srv1226843480=6, srv1341675059=10, srv1424862959=14, srv512572293=24, srv1097437830=2, srv1108495378=3, srv1385407332=12, srv1416325396=13, srv1920066092=18, srv906482856=29, srv1020936385=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-23 16:55:16,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-23 16:55:16,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-23 16:55:16,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-23 16:55:16,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-23 16:55:16,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-23 16:55:16,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-23 16:55:16,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-23 16:55:16,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-23 16:55:16,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-23 16:55:16,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-23 16:55:16,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-23 16:55:16,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-23 16:55:16,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-23 16:55:16,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-23 16:55:16,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-23 16:55:16,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-23 16:55:16,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-23 16:55:16,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-23 16:55:16,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-23 16:55:16,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-23 16:55:16,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-23 16:55:16,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-23 16:55:16,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-23 16:55:16,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-23 16:55:16,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-23 16:55:16,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-23 16:55:16,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-23 16:55:16,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-23 16:55:16,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-23 16:55:16,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-23 16:55:16,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-07-23 16:55:16,273 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-23 16:55:16,273 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1020936385:90 , srv1043849478:90 , srv1097437830:90 , srv1108495378:90 , srv1129914153:90 , srv120059140:90 , srv1226843480:90 , srv1230480169:90 , srv1234077708:90 , srv1247075104:90 , srv1341675059:90 , srv1352107377:90 , srv1385407332:90 , srv1416325396:90 , srv1424862959:90 , srv1438340768:90 , srv1635464456:90 , srv1709941459:90 , srv1920066092:90 , srv2099547683:90 , srv266477452:90 , srv418394859:90 , srv44353901:90 , srv453583932:90 , srv512572293:90 , srv556879481:90 , srv576829177:90 , srv688755869:90 , srv72145975:90 , srv906482856:90 } 2023-07-23 16:55:16,286 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=204 (was 294), ProcessCount=171 (was 175), AvailableMemoryMB=9646 (was 9799)