2023-09-29 18:53:14,542 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-09-29 18:53:14,890 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-09-29 18:53:15,000 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-09-29 18:53:15,073 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-29 18:53:15,081 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-09-29 18:53:15,094 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=337, ProcessCount=173, AvailableMemoryMB=3075 2023-09-29 18:53:15,096 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-29 18:53:15,097 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-09-29 18:53:15,239 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1021375238:84 , srv1027556680:84 , srv1031037267:84 , srv1050840619:84 , srv1059391668:84 , srv106555499:84 , srv108009430:84 , srv1105816003:84 , srv1131414369:84 , srv1158703280:84 , srv1159073399:84 , srv1166724379:84 , srv1219678224:84 , srv1234370065:84 , srv1239973199:84 , srv1248968816:84 , srv1291363044:84 , srv1300479945:84 , srv1307385961:84 , srv133022139:84 , srv1401811369:84 , srv143179845:84 , srv1438600105:84 , srv1466401255:84 , srv1466523673:84 , srv1490502285:84 , srv1503032397:84 , srv1548011386:84 , srv1551794395:84 , srv156931533:84 , srv1576984842:84 , srv1598883445:84 , srv1605111138:84 , srv1616626572:84 , srv161764254:84 , srv1646637883:84 , srv1663026599:84 , srv1717605804:84 , srv1722196748:84 , srv1741643139:84 , srv1751439914:84 , srv1761314474:84 , srv1768634958:84 , srv1789109719:84 , srv1791038217:84 , srv1829147672:84 , srv1837791374:84 , srv1851396601:84 , srv1862871946:84 , srv1901819344:84 , srv1902524040:84 , srv1936278810:84 , srv1939707115:84 , srv1948063536:84 , srv1953552633:84 , srv1983292708:84 , srv1994060697:84 , srv2014149170:84 , srv2015984377:84 , srv2060309947:84 , srv206165499:84 , srv2063798965:84 , srv2070333845:84 , srv2091891792:84 , srv2121985692:84 , srv2147006283:84 , srv278981938:84 , srv298085880:84 , srv306937601:84 , srv374705958:84 , srv409518644:84 , srv46143216:84 , srv466068448:84 , srv47760341:84 , srv506740978:84 , srv516550836:84 , srv55838186:84 , srv588832629:84 , srv591514192:84 , srv618935146:84 , srv661403382:84 , srv697626947:84 , srv708032801:84 , srv711787423:84 , srv714874213:84 , srv726457621:84 , srv726668490:84 , srv750090092:84 , srv778181808:84 , srv783087112:84 , srv789446153:84 , srv79729834:84 , srv809033402:84 , srv816896265:84 , srv924431661:84 , srv934807945:84 , srv937245074:84 , srv967332546:84 , srv979099494:84 , srv842451628:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-09-29 18:53:15,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-29 18:53:15,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1021375238=0, srv750090092=87, srv1598883445=31, srv374705958=69, srv1789109719=43, srv979099494=99, srv1466523673=24, srv1239973199=14, srv1646637883=35, srv108009430=6, srv1219678224=12, srv1438600105=22, srv1741643139=39, srv206165499=60, srv466068448=72, srv2015984377=58, srv789446153=90, srv1761314474=41, srv967332546=98, srv1027556680=1, srv711787423=83, srv714874213=84, srv409518644=70, srv79729834=91, srv591514192=78, srv1717605804=37, srv783087112=89, srv934807945=96, srv1829147672=45, srv1722196748=38, srv1503032397=26, srv47760341=73, srv2091891792=63, srv1751439914=40, srv516550836=75, srv1131414369=8, srv1490502285=25, srv1953552633=54, srv1166724379=11, srv1901819344=49, srv278981938=66, srv1576984842=30, srv1548011386=27, srv156931533=29, srv1994060697=56, srv1059391668=4, srv1605111138=32, srv816896265=93, srv1159073399=10, srv133022139=19, srv55838186=76, srv697626947=81, srv1939707115=52, srv1851396601=47, srv1401811369=20, srv161764254=34, srv1663026599=36, srv661403382=80, srv1234370065=13, srv1031037267=2, srv1307385961=18, srv1105816003=7, srv588832629=77, srv2070333845=62, srv924431661=95, srv1248968816=15, srv842451628=94, srv1300479945=17, srv143179845=21, srv298085880=67, srv618935146=79, srv1158703280=9, srv1791038217=44, srv1983292708=55, srv46143216=71, srv726457621=85, srv726668490=86, srv1466401255=23, srv1768634958=42, srv1050840619=3, srv306937601=68, srv1862871946=48, srv2060309947=59, srv2063798965=61, srv1902524040=50, srv809033402=92, srv1551794395=28, srv937245074=97, srv2147006283=65, srv1616626572=33, srv2121985692=64, srv708032801=82, srv106555499=5, srv1291363044=16, srv1936278810=51, srv506740978=74, srv2014149170=57, srv778181808=88, srv1948063536=53, srv1837791374=46} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-29 18:53:15,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-29 18:53:15,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-29 18:53:15,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-29 18:53:15,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-29 18:53:15,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-29 18:53:15,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-29 18:53:15,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-29 18:53:15,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-29 18:53:15,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-29 18:53:15,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-29 18:53:15,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-29 18:53:15,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-29 18:53:15,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-29 18:53:15,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-29 18:53:15,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-29 18:53:15,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-29 18:53:15,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-29 18:53:15,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-29 18:53:15,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-29 18:53:15,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-29 18:53:15,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-29 18:53:15,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-29 18:53:15,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-29 18:53:15,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-29 18:53:15,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-29 18:53:15,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-29 18:53:15,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-29 18:53:15,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-29 18:53:15,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-29 18:53:15,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-29 18:53:15,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-29 18:53:15,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-29 18:53:15,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-29 18:53:15,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-29 18:53:15,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-29 18:53:15,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-29 18:53:15,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-29 18:53:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-29 18:53:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-29 18:53:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-29 18:53:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-29 18:53:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-29 18:53:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-29 18:53:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-29 18:53:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-29 18:53:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-29 18:53:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-29 18:53:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-29 18:53:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-29 18:53:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-29 18:53:15,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-29 18:53:15,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-29 18:53:15,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-29 18:53:15,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-29 18:53:15,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-29 18:53:15,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-29 18:53:15,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-29 18:53:15,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-29 18:53:15,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-29 18:53:15,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-29 18:53:15,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-29 18:53:15,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-29 18:53:15,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-29 18:53:15,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-29 18:53:15,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-29 18:53:15,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-29 18:53:15,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-29 18:53:15,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-29 18:53:15,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-29 18:53:15,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-29 18:53:15,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-29 18:53:15,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-29 18:53:15,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-29 18:53:15,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-29 18:53:15,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-29 18:53:15,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-29 18:53:15,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-29 18:53:15,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-29 18:53:15,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-29 18:53:15,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-29 18:53:15,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-29 18:53:15,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-29 18:53:15,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-29 18:53:15,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-29 18:53:15,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-29 18:53:15,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-29 18:53:15,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-29 18:53:15,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-29 18:53:15,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-29 18:53:15,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-29 18:53:15,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-29 18:53:15,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-29 18:53:15,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-29 18:53:15,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-29 18:53:15,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-29 18:53:15,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-29 18:53:15,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-29 18:53:15,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-29 18:53:15,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-29 18:53:15,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-29 18:53:15,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-29 18:53:15,567 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-29 18:53:15,567 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-29 18:53:15,567 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-29 18:53:15,903 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 509 ms to try 5000 different iterations. Found a solution that moves 4648 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18646440282376298. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04072359602522949, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05800847494102189, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5164444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04072359602522949, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18191666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24658333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-29 18:53:15,920 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1105816003:82 , srv55838186:82 , srv937245074:82 , srv1158703280:83 , srv1239973199:83 , srv1490502285:83 , srv298085880:83 , srv697626947:83 , srv809033402:83 , srv1050840619:84 , srv106555499:84 , srv1307385961:84 , srv1438600105:84 , srv1616626572:84 , srv1789109719:84 , srv1791038217:84 , srv2063798965:84 , srv778181808:84 , srv783087112:84 , srv924431661:84 , srv1059391668:85 , srv108009430:85 , srv1159073399:85 , srv1166724379:85 , srv1219678224:85 , srv1466523673:85 , srv1503032397:85 , srv1598883445:85 , srv161764254:85 , srv1741643139:85 , srv1751439914:85 , srv1837791374:85 , srv1953552633:85 , srv2147006283:85 , srv374705958:85 , srv46143216:85 , srv466068448:85 , srv516550836:85 , srv708032801:85 , srv979099494:85 , srv1031037267:86 , srv1234370065:86 , srv1300479945:86 , srv1401811369:86 , srv1466401255:86 , srv156931533:86 , srv1605111138:86 , srv1722196748:86 , srv1862871946:86 , srv2060309947:86 , srv206165499:86 , srv278981938:86 , srv409518644:86 , srv506740978:86 , srv588832629:86 , srv618935146:86 , srv711787423:86 , srv726668490:86 , srv1021375238:87 , srv1027556680:87 , srv1131414369:87 , srv1576984842:87 , srv1663026599:87 , srv1901819344:87 , srv1939707115:87 , srv1948063536:87 , srv1994060697:87 , srv2014149170:87 , srv2121985692:87 , srv47760341:87 , srv591514192:87 , srv714874213:87 , srv726457621:87 , srv750090092:87 , srv79729834:87 , srv816896265:87 , srv1248968816:88 , srv133022139:88 , srv1761314474:88 , srv1829147672:88 , srv1936278810:88 , srv934807945:88 , srv1291363044:89 , srv1548011386:89 , srv1551794395:89 , srv1646637883:89 , srv1768634958:89 , srv1983292708:89 , srv2070333845:89 , srv661403382:89 , srv789446153:89 , srv2015984377:90 , srv306937601:90 , srv967332546:90 , srv2091891792:91 , srv143179845:92 , srv1851396601:93 , srv1902524040:94 , srv1717605804:97 , srv842451628:452 } 2023-09-29 18:53:15,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-29 18:53:15,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1021375238=0, srv750090092=87, srv1598883445=31, srv374705958=69, srv1789109719=43, srv979099494=99, srv1466523673=24, srv1239973199=14, srv1646637883=35, srv108009430=6, srv1219678224=12, srv1438600105=22, srv1741643139=39, srv206165499=60, srv466068448=72, srv2015984377=58, srv789446153=90, srv1761314474=41, srv967332546=98, srv1027556680=1, srv711787423=83, srv714874213=84, srv409518644=70, srv79729834=91, srv591514192=78, srv1717605804=37, srv783087112=89, srv934807945=96, srv1829147672=45, srv1722196748=38, srv1503032397=26, srv47760341=73, srv2091891792=63, srv1751439914=40, srv516550836=75, srv1131414369=8, srv1490502285=25, srv1953552633=54, srv1166724379=11, srv1901819344=49, srv278981938=66, srv1576984842=30, srv1548011386=27, srv156931533=29, srv1994060697=56, srv1059391668=4, srv1605111138=32, srv816896265=93, srv1159073399=10, srv133022139=19, srv55838186=76, srv697626947=81, srv1939707115=52, srv1851396601=47, srv1401811369=20, srv161764254=34, srv1663026599=36, srv661403382=80, srv1234370065=13, srv1031037267=2, srv1307385961=18, srv1105816003=7, srv588832629=77, srv2070333845=62, srv924431661=95, srv1248968816=15, srv842451628=94, srv1300479945=17, srv143179845=21, srv298085880=67, srv618935146=79, srv1158703280=9, srv1791038217=44, srv1983292708=55, srv46143216=71, srv726457621=85, srv726668490=86, srv1466401255=23, srv1768634958=42, srv1050840619=3, srv306937601=68, srv1862871946=48, srv2060309947=59, srv2063798965=61, srv1902524040=50, srv809033402=92, srv1551794395=28, srv937245074=97, srv2147006283=65, srv1616626572=33, srv2121985692=64, srv708032801=82, srv106555499=5, srv1291363044=16, srv1936278810=51, srv506740978=74, srv2014149170=57, srv778181808=88, srv1948063536=53, srv1837791374=46} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-29 18:53:15,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-29 18:53:15,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-29 18:53:15,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-29 18:53:15,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-29 18:53:15,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-29 18:53:15,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-29 18:53:15,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-29 18:53:15,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-29 18:53:15,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-29 18:53:15,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-29 18:53:15,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-29 18:53:15,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-29 18:53:15,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-29 18:53:15,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-29 18:53:15,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-29 18:53:15,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-29 18:53:15,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-29 18:53:15,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-29 18:53:15,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-29 18:53:15,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-29 18:53:15,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-29 18:53:15,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-29 18:53:15,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-29 18:53:15,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-29 18:53:15,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-29 18:53:15,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-29 18:53:15,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-29 18:53:15,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-29 18:53:15,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-29 18:53:15,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-29 18:53:15,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-29 18:53:15,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-29 18:53:15,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-29 18:53:15,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-29 18:53:15,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-29 18:53:15,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-29 18:53:15,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-29 18:53:15,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-29 18:53:15,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-29 18:53:15,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-29 18:53:15,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-29 18:53:15,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-29 18:53:15,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-29 18:53:15,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-29 18:53:15,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-29 18:53:15,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-29 18:53:15,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-29 18:53:15,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-29 18:53:15,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-29 18:53:15,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-29 18:53:15,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-29 18:53:15,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-29 18:53:15,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-29 18:53:15,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-29 18:53:15,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-29 18:53:15,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-29 18:53:15,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-29 18:53:15,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-29 18:53:15,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-29 18:53:15,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-29 18:53:15,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-29 18:53:15,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-29 18:53:15,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-29 18:53:15,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-29 18:53:15,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-29 18:53:15,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-29 18:53:15,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-29 18:53:15,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-29 18:53:15,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-29 18:53:15,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-29 18:53:15,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-29 18:53:15,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-29 18:53:15,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-29 18:53:15,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-29 18:53:15,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-29 18:53:15,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-29 18:53:15,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-29 18:53:15,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-29 18:53:15,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-29 18:53:15,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-29 18:53:15,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-29 18:53:15,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-29 18:53:15,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-29 18:53:15,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-29 18:53:15,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-29 18:53:15,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-29 18:53:15,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-29 18:53:15,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-29 18:53:15,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-29 18:53:15,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-29 18:53:15,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-29 18:53:15,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-29 18:53:15,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-29 18:53:15,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-29 18:53:15,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-29 18:53:15,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-29 18:53:15,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-29 18:53:15,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-29 18:53:15,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-29 18:53:15,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-29 18:53:15,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-29 18:53:16,138 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04072359602522949, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05800847494102189, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04072359602522949, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18191666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24658333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-29 18:53:16,139 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-29 18:53:16,139 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18643185683214356, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04072359602522949, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05800847494102189, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04072359602522949, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18191666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24658333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-29 18:53:16,448 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 451 ms to try 5000 different iterations. Found a solution that moves 2668 regions; Going from a computed imbalance of 0.18643185683214356 to a new imbalance of 0.015989233617726015. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023056180944285368); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034918535946363585, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29644444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023056180944285368); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-29 18:53:16,457 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1717605804:77 , srv1902524040:77 , srv1851396601:79 , srv143179845:80 , srv2091891792:81 , srv967332546:81 , srv1829147672:82 , srv2015984377:82 , srv306937601:82 , srv1291363044:83 , srv1768634958:83 , srv1983292708:83 , srv2070333845:83 , srv2121985692:83 , srv789446153:83 , srv79729834:83 , srv1031037267:84 , srv1131414369:84 , srv1234370065:84 , srv133022139:84 , srv1548011386:84 , srv156931533:84 , srv1646637883:84 , srv1663026599:84 , srv1761314474:84 , srv506740978:84 , srv588832629:84 , srv591514192:84 , srv1027556680:85 , srv1248968816:85 , srv1300479945:85 , srv1466523673:85 , srv1551794395:85 , srv1576984842:85 , srv1751439914:85 , srv1862871946:85 , srv1901819344:85 , srv206165499:85 , srv278981938:85 , srv409518644:85 , srv816896265:85 , srv934807945:85 , srv1021375238:86 , srv1059391668:86 , srv1166724379:86 , srv1307385961:86 , srv1401811369:86 , srv1466401255:86 , srv1722196748:86 , srv1789109719:86 , srv1939707115:86 , srv1953552633:86 , srv1994060697:86 , srv2014149170:86 , srv2060309947:86 , srv466068448:86 , srv661403382:86 , srv726457621:86 , srv726668490:86 , srv750090092:86 , srv106555499:87 , srv108009430:87 , srv1159073399:87 , srv1219678224:87 , srv1503032397:87 , srv1598883445:87 , srv1605111138:87 , srv161764254:87 , srv1741643139:87 , srv1791038217:87 , srv1837791374:87 , srv1948063536:87 , srv2063798965:87 , srv2147006283:87 , srv298085880:87 , srv374705958:87 , srv47760341:87 , srv516550836:87 , srv697626947:87 , srv711787423:87 , srv783087112:87 , srv1490502285:88 , srv1936278810:88 , srv46143216:88 , srv55838186:88 , srv618935146:88 , srv714874213:88 , srv778181808:88 , srv924431661:88 , srv937245074:88 , srv1105816003:89 , srv1239973199:89 , srv1438600105:89 , srv1616626572:89 , srv708032801:89 , srv809033402:89 , srv1050840619:90 , srv979099494:90 , srv1158703280:92 , srv842451628:527 } 2023-09-29 18:53:16,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-29 18:53:16,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1021375238=0, srv750090092=87, srv1598883445=31, srv374705958=69, srv1789109719=43, srv979099494=99, srv1466523673=24, srv1239973199=14, srv1646637883=35, srv108009430=6, srv1219678224=12, srv1438600105=22, srv1741643139=39, srv206165499=60, srv466068448=72, srv2015984377=58, srv789446153=90, srv1761314474=41, srv967332546=98, srv1027556680=1, srv711787423=83, srv714874213=84, srv409518644=70, srv79729834=91, srv591514192=78, srv1717605804=37, srv783087112=89, srv934807945=96, srv1829147672=45, srv1722196748=38, srv1503032397=26, srv47760341=73, srv2091891792=63, srv1751439914=40, srv516550836=75, srv1131414369=8, srv1490502285=25, srv1953552633=54, srv1166724379=11, srv1901819344=49, srv278981938=66, srv1576984842=30, srv1548011386=27, srv156931533=29, srv1994060697=56, srv1059391668=4, srv1605111138=32, srv816896265=93, srv1159073399=10, srv133022139=19, srv55838186=76, srv697626947=81, srv1939707115=52, srv1851396601=47, srv1401811369=20, srv161764254=34, srv1663026599=36, srv661403382=80, srv1234370065=13, srv1031037267=2, srv1307385961=18, srv1105816003=7, srv588832629=77, srv2070333845=62, srv924431661=95, srv1248968816=15, srv842451628=94, srv1300479945=17, srv143179845=21, srv298085880=67, srv618935146=79, srv1158703280=9, srv1791038217=44, srv1983292708=55, srv46143216=71, srv726457621=85, srv726668490=86, srv1466401255=23, srv1768634958=42, srv1050840619=3, srv306937601=68, srv1862871946=48, srv2060309947=59, srv2063798965=61, srv1902524040=50, srv809033402=92, srv1551794395=28, srv937245074=97, srv2147006283=65, srv1616626572=33, srv2121985692=64, srv708032801=82, srv106555499=5, srv1291363044=16, srv1936278810=51, srv506740978=74, srv2014149170=57, srv778181808=88, srv1948063536=53, srv1837791374=46} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-29 18:53:16,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-29 18:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-29 18:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-29 18:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-29 18:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-29 18:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-29 18:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-29 18:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-29 18:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-29 18:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-29 18:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-29 18:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-29 18:53:16,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-29 18:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-29 18:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-29 18:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-29 18:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-29 18:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-29 18:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-29 18:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-29 18:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-29 18:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-29 18:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-29 18:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-29 18:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-29 18:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-29 18:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-29 18:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-29 18:53:16,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-29 18:53:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-29 18:53:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-29 18:53:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-29 18:53:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-29 18:53:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-29 18:53:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-29 18:53:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-29 18:53:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-29 18:53:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-29 18:53:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-29 18:53:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-29 18:53:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-29 18:53:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-29 18:53:16,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-29 18:53:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-29 18:53:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-29 18:53:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-29 18:53:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-29 18:53:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-29 18:53:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-29 18:53:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-29 18:53:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-29 18:53:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-29 18:53:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-29 18:53:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-29 18:53:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-29 18:53:16,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-29 18:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-29 18:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-29 18:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-29 18:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-29 18:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-29 18:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-29 18:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-29 18:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-29 18:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-29 18:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-29 18:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-29 18:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-29 18:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-29 18:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-29 18:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-29 18:53:16,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-29 18:53:16,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-29 18:53:16,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-29 18:53:16,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-29 18:53:16,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-29 18:53:16,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-29 18:53:16,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-29 18:53:16,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-29 18:53:16,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-29 18:53:16,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-29 18:53:16,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-29 18:53:16,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-29 18:53:16,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-29 18:53:16,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-29 18:53:16,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-29 18:53:16,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-29 18:53:16,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-29 18:53:16,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-29 18:53:16,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-29 18:53:16,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-29 18:53:16,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-29 18:53:16,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-29 18:53:16,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-29 18:53:16,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-29 18:53:16,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-29 18:53:16,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-29 18:53:16,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-29 18:53:16,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-29 18:53:16,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-29 18:53:16,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-29 18:53:16,616 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023056180944285368); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034918535946363585, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023056180944285368); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-29 18:53:16,616 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-29 18:53:16,616 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01597055188243328, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023056180944285368); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034918535946363585, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023056180944285368); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-29 18:53:17,051 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 546 ms to try 5000 different iterations. Found a solution that moves 997 regions; Going from a computed imbalance of 0.01597055188243328 to a new imbalance of 5.580031554503178E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007778579403935822); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00908866414480969); MoveCostFunction : (multiplier=7.0, imbalance=0.11077777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007778579403935822); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-29 18:53:17,058 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1936278810:81 , srv1158703280:82 , srv714874213:82 , srv1050840619:83 , srv1548011386:83 , srv1551794395:83 , srv1994060697:83 , srv47760341:83 , srv661403382:83 , srv708032801:83 , srv979099494:83 , srv108009430:84 , srv1248968816:84 , srv1438600105:84 , srv1466401255:84 , srv1646637883:84 , srv1717605804:84 , srv1939707115:84 , srv1948063536:84 , srv2014149170:84 , srv46143216:84 , srv516550836:84 , srv618935146:84 , srv726457621:84 , srv750090092:84 , srv789446153:84 , srv1021375238:85 , srv1027556680:85 , srv1159073399:85 , srv1219678224:85 , srv1239973199:85 , srv133022139:85 , srv143179845:85 , srv1503032397:85 , srv1605111138:85 , srv1616626572:85 , srv1722196748:85 , srv1761314474:85 , srv1768634958:85 , srv1837791374:85 , srv1901819344:85 , srv2015984377:85 , srv2070333845:85 , srv2091891792:85 , srv306937601:85 , srv374705958:85 , srv466068448:85 , srv711787423:85 , srv726668490:85 , srv778181808:85 , srv783087112:85 , srv924431661:85 , srv934807945:85 , srv967332546:85 , srv1059391668:86 , srv106555499:86 , srv1131414369:86 , srv1291363044:86 , srv1401811369:86 , srv1576984842:86 , srv1598883445:86 , srv161764254:86 , srv1663026599:86 , srv1741643139:86 , srv1851396601:86 , srv1862871946:86 , srv1953552633:86 , srv1983292708:86 , srv206165499:86 , srv2147006283:86 , srv591514192:86 , srv809033402:86 , srv816896265:86 , srv1166724379:87 , srv1234370065:87 , srv1300479945:87 , srv1307385961:87 , srv1466523673:87 , srv1490502285:87 , srv156931533:87 , srv1751439914:87 , srv1789109719:87 , srv1791038217:87 , srv1902524040:87 , srv2060309947:87 , srv2063798965:87 , srv2121985692:87 , srv278981938:87 , srv298085880:87 , srv409518644:87 , srv506740978:87 , srv937245074:87 , srv1031037267:88 , srv1105816003:88 , srv1829147672:88 , srv55838186:88 , srv588832629:88 , srv697626947:88 , srv79729834:88 , srv842451628:548 } 2023-09-29 18:53:17,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-29 18:53:17,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1021375238=0, srv750090092=87, srv1598883445=31, srv374705958=69, srv1789109719=43, srv979099494=99, srv1466523673=24, srv1239973199=14, srv1646637883=35, srv108009430=6, srv1219678224=12, srv1438600105=22, srv1741643139=39, srv206165499=60, srv466068448=72, srv2015984377=58, srv789446153=90, srv1761314474=41, srv967332546=98, srv1027556680=1, srv711787423=83, srv714874213=84, srv409518644=70, srv79729834=91, srv591514192=78, srv1717605804=37, srv783087112=89, srv934807945=96, srv1829147672=45, srv1722196748=38, srv1503032397=26, srv47760341=73, srv2091891792=63, srv1751439914=40, srv516550836=75, srv1131414369=8, srv1490502285=25, srv1953552633=54, srv1166724379=11, srv1901819344=49, srv278981938=66, srv1576984842=30, srv1548011386=27, srv156931533=29, srv1994060697=56, srv1059391668=4, srv1605111138=32, srv816896265=93, srv1159073399=10, srv133022139=19, srv55838186=76, srv697626947=81, srv1939707115=52, srv1851396601=47, srv1401811369=20, srv161764254=34, srv1663026599=36, srv661403382=80, srv1234370065=13, srv1031037267=2, srv1307385961=18, srv1105816003=7, srv588832629=77, srv2070333845=62, srv924431661=95, srv1248968816=15, srv842451628=94, srv1300479945=17, srv143179845=21, srv298085880=67, srv618935146=79, srv1158703280=9, srv1791038217=44, srv1983292708=55, srv46143216=71, srv726457621=85, srv726668490=86, srv1466401255=23, srv1768634958=42, srv1050840619=3, srv306937601=68, srv1862871946=48, srv2060309947=59, srv2063798965=61, srv1902524040=50, srv809033402=92, srv1551794395=28, srv937245074=97, srv2147006283=65, srv1616626572=33, srv2121985692=64, srv708032801=82, srv106555499=5, srv1291363044=16, srv1936278810=51, srv506740978=74, srv2014149170=57, srv778181808=88, srv1948063536=53, srv1837791374=46} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-29 18:53:17,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-29 18:53:17,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-29 18:53:17,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-29 18:53:17,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-29 18:53:17,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-29 18:53:17,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-29 18:53:17,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-29 18:53:17,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-29 18:53:17,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-29 18:53:17,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-29 18:53:17,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-29 18:53:17,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-29 18:53:17,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-29 18:53:17,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-29 18:53:17,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-29 18:53:17,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-29 18:53:17,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-29 18:53:17,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-29 18:53:17,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-29 18:53:17,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-29 18:53:17,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-29 18:53:17,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-29 18:53:17,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-29 18:53:17,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-29 18:53:17,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-29 18:53:17,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-29 18:53:17,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-29 18:53:17,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-29 18:53:17,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-29 18:53:17,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-29 18:53:17,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-29 18:53:17,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-29 18:53:17,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-29 18:53:17,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-29 18:53:17,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-29 18:53:17,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-29 18:53:17,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-29 18:53:17,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-29 18:53:17,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-29 18:53:17,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-29 18:53:17,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-29 18:53:17,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-29 18:53:17,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-29 18:53:17,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-29 18:53:17,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-29 18:53:17,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-29 18:53:17,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-29 18:53:17,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-29 18:53:17,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-29 18:53:17,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-29 18:53:17,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-29 18:53:17,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-29 18:53:17,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-29 18:53:17,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-29 18:53:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-29 18:53:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-29 18:53:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-29 18:53:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-29 18:53:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-29 18:53:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-29 18:53:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-29 18:53:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-29 18:53:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-29 18:53:17,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-29 18:53:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-29 18:53:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-29 18:53:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-29 18:53:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-29 18:53:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-29 18:53:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-29 18:53:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-29 18:53:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-29 18:53:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-29 18:53:17,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-29 18:53:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-29 18:53:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-29 18:53:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-29 18:53:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-29 18:53:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-29 18:53:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-29 18:53:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-29 18:53:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-29 18:53:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-29 18:53:17,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-29 18:53:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-29 18:53:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-29 18:53:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-29 18:53:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-29 18:53:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-29 18:53:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-29 18:53:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-29 18:53:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-29 18:53:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-29 18:53:17,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-29 18:53:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-29 18:53:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-29 18:53:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-29 18:53:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-29 18:53:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-29 18:53:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-29 18:53:17,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-29 18:53:17,300 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007778579403935822); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00908866414480969); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007778579403935822); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-29 18:53:17,300 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-29 18:53:17,301 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.510220122393522E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007778579403935822); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00908866414480969); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007778579403935822); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-29 18:53:17,648 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 501 ms to try 5000 different iterations. Found a solution that moves 249 regions; Going from a computed imbalance of 5.510220122393522E-4 to a new imbalance of 3.57039476253728E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.027666666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-29 18:53:17,650 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1105816003:83 , srv409518644:83 , srv1021375238:84 , srv1031037267:84 , srv1059391668:84 , srv1166724379:84 , srv1291363044:84 , srv1300479945:84 , srv1401811369:84 , srv1438600105:84 , srv1490502285:84 , srv1503032397:84 , srv1551794395:84 , srv1576984842:84 , srv1598883445:84 , srv1605111138:84 , srv1616626572:84 , srv161764254:84 , srv1717605804:84 , srv1741643139:84 , srv1761314474:84 , srv1791038217:84 , srv1829147672:84 , srv1851396601:84 , srv1902524040:84 , srv1948063536:84 , srv1983292708:84 , srv2060309947:84 , srv2063798965:84 , srv2147006283:84 , srv278981938:84 , srv55838186:84 , srv588832629:84 , srv661403382:84 , srv711787423:84 , srv816896265:84 , srv934807945:84 , srv979099494:84 , srv1027556680:85 , srv1050840619:85 , srv106555499:85 , srv1131414369:85 , srv1158703280:85 , srv1159073399:85 , srv1234370065:85 , srv1239973199:85 , srv1248968816:85 , srv133022139:85 , srv143179845:85 , srv1466401255:85 , srv1466523673:85 , srv1548011386:85 , srv156931533:85 , srv1646637883:85 , srv1722196748:85 , srv1751439914:85 , srv1768634958:85 , srv1789109719:85 , srv1837791374:85 , srv1862871946:85 , srv1901819344:85 , srv1936278810:85 , srv1939707115:85 , srv1953552633:85 , srv1994060697:85 , srv2014149170:85 , srv2015984377:85 , srv206165499:85 , srv2070333845:85 , srv2091891792:85 , srv2121985692:85 , srv298085880:85 , srv306937601:85 , srv374705958:85 , srv46143216:85 , srv466068448:85 , srv47760341:85 , srv506740978:85 , srv591514192:85 , srv618935146:85 , srv697626947:85 , srv708032801:85 , srv714874213:85 , srv726457621:85 , srv726668490:85 , srv778181808:85 , srv79729834:85 , srv809033402:85 , srv924431661:85 , srv937245074:85 , srv108009430:86 , srv1219678224:86 , srv1307385961:86 , srv1663026599:86 , srv516550836:86 , srv750090092:86 , srv783087112:86 , srv789446153:86 , srv967332546:86 , srv842451628:616 } 2023-09-29 18:53:17,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-29 18:53:17,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1021375238=0, srv750090092=87, srv1598883445=31, srv374705958=69, srv1789109719=43, srv979099494=99, srv1466523673=24, srv1239973199=14, srv1646637883=35, srv108009430=6, srv1219678224=12, srv1438600105=22, srv1741643139=39, srv206165499=60, srv466068448=72, srv2015984377=58, srv789446153=90, srv1761314474=41, srv967332546=98, srv1027556680=1, srv711787423=83, srv714874213=84, srv409518644=70, srv79729834=91, srv591514192=78, srv1717605804=37, srv783087112=89, srv934807945=96, srv1829147672=45, srv1722196748=38, srv1503032397=26, srv47760341=73, srv2091891792=63, srv1751439914=40, srv516550836=75, srv1131414369=8, srv1490502285=25, srv1953552633=54, srv1166724379=11, srv1901819344=49, srv278981938=66, srv1576984842=30, srv1548011386=27, srv156931533=29, srv1994060697=56, srv1059391668=4, srv1605111138=32, srv816896265=93, srv1159073399=10, srv133022139=19, srv55838186=76, srv697626947=81, srv1939707115=52, srv1851396601=47, srv1401811369=20, srv161764254=34, srv1663026599=36, srv661403382=80, srv1234370065=13, srv1031037267=2, srv1307385961=18, srv1105816003=7, srv588832629=77, srv2070333845=62, srv924431661=95, srv1248968816=15, srv842451628=94, srv1300479945=17, srv143179845=21, srv298085880=67, srv618935146=79, srv1158703280=9, srv1791038217=44, srv1983292708=55, srv46143216=71, srv726457621=85, srv726668490=86, srv1466401255=23, srv1768634958=42, srv1050840619=3, srv306937601=68, srv1862871946=48, srv2060309947=59, srv2063798965=61, srv1902524040=50, srv809033402=92, srv1551794395=28, srv937245074=97, srv2147006283=65, srv1616626572=33, srv2121985692=64, srv708032801=82, srv106555499=5, srv1291363044=16, srv1936278810=51, srv506740978=74, srv2014149170=57, srv778181808=88, srv1948063536=53, srv1837791374=46} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-29 18:53:17,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-29 18:53:17,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-29 18:53:17,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-29 18:53:17,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-29 18:53:17,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-29 18:53:17,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-29 18:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-29 18:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-29 18:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-29 18:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-29 18:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-29 18:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-29 18:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-29 18:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-29 18:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-29 18:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-29 18:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-29 18:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-29 18:53:17,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-29 18:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-29 18:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-29 18:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-29 18:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-29 18:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-29 18:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-29 18:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-29 18:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-29 18:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-29 18:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-29 18:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-29 18:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-29 18:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-29 18:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-29 18:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-29 18:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-29 18:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-29 18:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-29 18:53:17,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-29 18:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-29 18:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-29 18:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-29 18:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-29 18:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-29 18:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-29 18:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-29 18:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-29 18:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-29 18:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-29 18:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-29 18:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-29 18:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-29 18:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-29 18:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-29 18:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-29 18:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-29 18:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-29 18:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-29 18:53:17,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-29 18:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-29 18:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-29 18:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-29 18:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-29 18:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-29 18:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-29 18:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-29 18:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-29 18:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-29 18:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-29 18:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-29 18:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-29 18:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-29 18:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-29 18:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-29 18:53:17,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-29 18:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-29 18:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-29 18:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-29 18:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-29 18:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-29 18:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-29 18:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-29 18:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-29 18:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-29 18:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-29 18:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-29 18:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-29 18:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-29 18:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-29 18:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-29 18:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-29 18:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-29 18:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-29 18:53:17,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-29 18:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-29 18:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-29 18:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-29 18:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-29 18:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-29 18:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-29 18:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-29 18:53:17,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-29 18:53:17,780 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-29 18:53:17,780 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-29 18:53:17,781 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.396041236004634E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-29 18:53:18,008 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 323 ms to try 5000 different iterations. Found a solution that moves 26 regions; Going from a computed imbalance of 3.396041236004634E-5 to a new imbalance of 8.161746008199175E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0028888888888888888); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-29 18:53:18,008 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1219678224:83 , srv1307385961:83 , srv1663026599:83 , srv2060309947:83 , srv618935146:83 , srv697626947:83 , srv750090092:83 , srv79729834:83 , srv809033402:83 , srv1021375238:84 , srv1027556680:84 , srv1031037267:84 , srv1050840619:84 , srv106555499:84 , srv108009430:84 , srv1105816003:84 , srv1131414369:84 , srv1158703280:84 , srv1159073399:84 , srv1166724379:84 , srv1234370065:84 , srv1239973199:84 , srv1248968816:84 , srv1291363044:84 , srv1300479945:84 , srv133022139:84 , srv1401811369:84 , srv143179845:84 , srv1466523673:84 , srv1490502285:84 , srv156931533:84 , srv1576984842:84 , srv1598883445:84 , srv1605111138:84 , srv1616626572:84 , srv161764254:84 , srv1646637883:84 , srv1717605804:84 , srv1722196748:84 , srv1741643139:84 , srv1751439914:84 , srv1768634958:84 , srv1789109719:84 , srv1791038217:84 , srv1829147672:84 , srv1837791374:84 , srv1851396601:84 , srv1862871946:84 , srv1901819344:84 , srv1902524040:84 , srv1936278810:84 , srv1939707115:84 , srv1948063536:84 , srv1953552633:84 , srv1983292708:84 , srv2014149170:84 , srv2015984377:84 , srv206165499:84 , srv2063798965:84 , srv2070333845:84 , srv2091891792:84 , srv2121985692:84 , srv2147006283:84 , srv278981938:84 , srv298085880:84 , srv306937601:84 , srv374705958:84 , srv46143216:84 , srv47760341:84 , srv506740978:84 , srv516550836:84 , srv55838186:84 , srv588832629:84 , srv591514192:84 , srv661403382:84 , srv708032801:84 , srv711787423:84 , srv714874213:84 , srv726457621:84 , srv726668490:84 , srv778181808:84 , srv783087112:84 , srv789446153:84 , srv816896265:84 , srv924431661:84 , srv934807945:84 , srv937245074:84 , srv967332546:84 , srv979099494:84 , srv1059391668:85 , srv1438600105:85 , srv1466401255:85 , srv1503032397:85 , srv1548011386:85 , srv1551794395:85 , srv1761314474:85 , srv1994060697:85 , srv409518644:85 , srv466068448:85 , srv842451628:683 } 2023-09-29 18:53:18,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-29 18:53:18,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1021375238=0, srv750090092=87, srv1598883445=31, srv374705958=69, srv1789109719=43, srv979099494=99, srv1466523673=24, srv1239973199=14, srv1646637883=35, srv108009430=6, srv1219678224=12, srv1438600105=22, srv1741643139=39, srv206165499=60, srv466068448=72, srv2015984377=58, srv789446153=90, srv1761314474=41, srv967332546=98, srv1027556680=1, srv711787423=83, srv714874213=84, srv409518644=70, srv79729834=91, srv591514192=78, srv1717605804=37, srv783087112=89, srv934807945=96, srv1829147672=45, srv1722196748=38, srv1503032397=26, srv47760341=73, srv2091891792=63, srv1751439914=40, srv516550836=75, srv1131414369=8, srv1490502285=25, srv1953552633=54, srv1166724379=11, srv1901819344=49, srv278981938=66, srv1576984842=30, srv1548011386=27, srv156931533=29, srv1994060697=56, srv1059391668=4, srv1605111138=32, srv816896265=93, srv1159073399=10, srv133022139=19, srv55838186=76, srv697626947=81, srv1939707115=52, srv1851396601=47, srv1401811369=20, srv161764254=34, srv1663026599=36, srv661403382=80, srv1234370065=13, srv1031037267=2, srv1307385961=18, srv1105816003=7, srv588832629=77, srv2070333845=62, srv924431661=95, srv1248968816=15, srv842451628=94, srv1300479945=17, srv143179845=21, srv298085880=67, srv618935146=79, srv1158703280=9, srv1791038217=44, srv1983292708=55, srv46143216=71, srv726457621=85, srv726668490=86, srv1466401255=23, srv1768634958=42, srv1050840619=3, srv306937601=68, srv1862871946=48, srv2060309947=59, srv2063798965=61, srv1902524040=50, srv809033402=92, srv1551794395=28, srv937245074=97, srv2147006283=65, srv1616626572=33, srv2121985692=64, srv708032801=82, srv106555499=5, srv1291363044=16, srv1936278810=51, srv506740978=74, srv2014149170=57, srv778181808=88, srv1948063536=53, srv1837791374=46} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-29 18:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-29 18:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-29 18:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-29 18:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-29 18:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-29 18:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-29 18:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-29 18:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-29 18:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-29 18:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-29 18:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-29 18:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-29 18:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-29 18:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-29 18:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-29 18:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-29 18:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-29 18:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-29 18:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-29 18:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-29 18:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-29 18:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-29 18:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-29 18:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-29 18:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-29 18:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-29 18:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-29 18:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-29 18:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-29 18:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-29 18:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-29 18:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-29 18:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-29 18:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-29 18:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-29 18:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-29 18:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-29 18:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-29 18:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-29 18:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-29 18:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-29 18:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-29 18:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-29 18:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-29 18:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-29 18:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-29 18:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-29 18:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-29 18:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-29 18:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-29 18:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-29 18:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-29 18:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-29 18:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-29 18:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-29 18:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-29 18:53:18,130 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.979690117040589E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-29 18:53:18,130 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1219678224:83 , srv1307385961:83 , srv1663026599:83 , srv2060309947:83 , srv618935146:83 , srv697626947:83 , srv750090092:83 , srv79729834:83 , srv809033402:83 , srv1021375238:84 , srv1027556680:84 , srv1031037267:84 , srv1050840619:84 , srv106555499:84 , srv108009430:84 , srv1105816003:84 , srv1131414369:84 , srv1158703280:84 , srv1159073399:84 , srv1166724379:84 , srv1234370065:84 , srv1239973199:84 , srv1248968816:84 , srv1291363044:84 , srv1300479945:84 , srv133022139:84 , srv1401811369:84 , srv143179845:84 , srv1466523673:84 , srv1490502285:84 , srv156931533:84 , srv1576984842:84 , srv1598883445:84 , srv1605111138:84 , srv1616626572:84 , srv161764254:84 , srv1646637883:84 , srv1717605804:84 , srv1722196748:84 , srv1741643139:84 , srv1751439914:84 , srv1768634958:84 , srv1789109719:84 , srv1791038217:84 , srv1829147672:84 , srv1837791374:84 , srv1851396601:84 , srv1862871946:84 , srv1901819344:84 , srv1902524040:84 , srv1936278810:84 , srv1939707115:84 , srv1948063536:84 , srv1953552633:84 , srv1983292708:84 , srv2014149170:84 , srv2015984377:84 , srv206165499:84 , srv2063798965:84 , srv2070333845:84 , srv2091891792:84 , srv2121985692:84 , srv2147006283:84 , srv278981938:84 , srv298085880:84 , srv306937601:84 , srv374705958:84 , srv46143216:84 , srv47760341:84 , srv506740978:84 , srv516550836:84 , srv55838186:84 , srv588832629:84 , srv591514192:84 , srv661403382:84 , srv708032801:84 , srv711787423:84 , srv714874213:84 , srv726457621:84 , srv726668490:84 , srv778181808:84 , srv783087112:84 , srv789446153:84 , srv816896265:84 , srv924431661:84 , srv934807945:84 , srv937245074:84 , srv967332546:84 , srv979099494:84 , srv1059391668:85 , srv1438600105:85 , srv1466401255:85 , srv1503032397:85 , srv1548011386:85 , srv1551794395:85 , srv1761314474:85 , srv1994060697:85 , srv409518644:85 , srv466068448:85 , srv842451628:683 } 2023-09-29 18:53:18,154 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=337 (was 337), ProcessCount=173 (was 173), AvailableMemoryMB=2459 (was 3075) 2023-09-29 18:53:18,162 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=337, ProcessCount=173, AvailableMemoryMB=2459 2023-09-29 18:53:18,163 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-29 18:53:18,163 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-09-29 18:53:18,175 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1319268853:84 , srv1379298556:84 , srv1468599215:84 , srv1608856220:84 , srv1629569681:84 , srv1643787057:84 , srv1721379399:84 , srv1732791035:84 , srv1748970366:84 , srv1780339900:84 , srv1912024631:84 , srv1935888120:84 , srv1936738579:84 , srv239281851:84 , srv279616799:84 , srv325852317:84 , srv337715421:84 , srv429753380:84 , srv512745593:84 , srv520412030:84 , srv521446265:84 , srv540513951:84 , srv643704247:84 , srv717855747:84 , srv782426224:84 , srv820862262:84 , srv904070720:84 , srv90852733:84 , srv988341284:84 , srv1654967001:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-09-29 18:53:18,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-29 18:53:18,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1935888120=12, srv904070720=27, srv1608856220=3, srv1732791035=8, srv1780339900=10, srv325852317=16, srv782426224=25, srv643704247=23, srv429753380=18, srv512745593=19, srv1468599215=2, srv1936738579=13, srv337715421=17, srv520412030=20, srv1748970366=9, srv1912024631=11, srv1721379399=7, srv820862262=26, srv1379298556=1, srv1654967001=6, srv279616799=15, srv90852733=28, srv239281851=14, srv521446265=21, srv1643787057=5, srv540513951=22, srv988341284=29, srv1319268853=0, srv1629569681=4, srv717855747=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-29 18:53:18,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-29 18:53:18,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-29 18:53:18,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-29 18:53:18,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-29 18:53:18,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-29 18:53:18,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-29 18:53:18,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-29 18:53:18,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-29 18:53:18,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-29 18:53:18,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-29 18:53:18,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-29 18:53:18,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-29 18:53:18,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-29 18:53:18,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-29 18:53:18,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-29 18:53:18,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-29 18:53:18,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-29 18:53:18,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-29 18:53:18,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-29 18:53:18,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-29 18:53:18,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-29 18:53:18,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-29 18:53:18,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-29 18:53:18,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-29 18:53:18,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-29 18:53:18,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-29 18:53:18,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-29 18:53:18,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-29 18:53:18,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-29 18:53:18,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-29 18:53:18,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-09-29 18:53:18,210 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0626096103162495, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-29 18:53:18,210 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912390324101996, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0626096103162495, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-09-29 18:55:18,198 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6068440 different iterations. Found a solution that moves 1856 regions; Going from a computed imbalance of 0.9912390324101996 to a new imbalance of 4.3319965896196796E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6874074074074074, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-29 18:55:18,203 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1319268853:90 , srv1379298556:90 , srv1468599215:90 , srv1608856220:90 , srv1629569681:90 , srv1643787057:90 , srv1654967001:90 , srv1721379399:90 , srv1732791035:90 , srv1748970366:90 , srv1780339900:90 , srv1912024631:90 , srv1935888120:90 , srv1936738579:90 , srv239281851:90 , srv279616799:90 , srv325852317:90 , srv337715421:90 , srv429753380:90 , srv512745593:90 , srv520412030:90 , srv521446265:90 , srv540513951:90 , srv643704247:90 , srv717855747:90 , srv782426224:90 , srv820862262:90 , srv904070720:90 , srv90852733:90 , srv988341284:90 } 2023-09-29 18:55:18,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-29 18:55:18,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1935888120=12, srv904070720=27, srv1608856220=3, srv1732791035=8, srv1780339900=10, srv325852317=16, srv782426224=25, srv643704247=23, srv429753380=18, srv512745593=19, srv1468599215=2, srv1936738579=13, srv337715421=17, srv520412030=20, srv1748970366=9, srv1912024631=11, srv1721379399=7, srv820862262=26, srv1379298556=1, srv1654967001=6, srv279616799=15, srv90852733=28, srv239281851=14, srv521446265=21, srv1643787057=5, srv540513951=22, srv988341284=29, srv1319268853=0, srv1629569681=4, srv717855747=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-29 18:55:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-29 18:55:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-29 18:55:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-29 18:55:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-29 18:55:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-29 18:55:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-29 18:55:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-29 18:55:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-29 18:55:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-29 18:55:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-29 18:55:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-29 18:55:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-29 18:55:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-29 18:55:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-29 18:55:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-29 18:55:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-29 18:55:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-29 18:55:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-29 18:55:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-29 18:55:18,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-29 18:55:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-29 18:55:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-29 18:55:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-29 18:55:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-29 18:55:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-29 18:55:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-29 18:55:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-29 18:55:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-29 18:55:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-29 18:55:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-29 18:55:18,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-09-29 18:55:18,221 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-29 18:55:18,221 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1319268853:90 , srv1379298556:90 , srv1468599215:90 , srv1608856220:90 , srv1629569681:90 , srv1643787057:90 , srv1654967001:90 , srv1721379399:90 , srv1732791035:90 , srv1748970366:90 , srv1780339900:90 , srv1912024631:90 , srv1935888120:90 , srv1936738579:90 , srv239281851:90 , srv279616799:90 , srv325852317:90 , srv337715421:90 , srv429753380:90 , srv512745593:90 , srv520412030:90 , srv521446265:90 , srv540513951:90 , srv643704247:90 , srv717855747:90 , srv782426224:90 , srv820862262:90 , srv904070720:90 , srv90852733:90 , srv988341284:90 } 2023-09-29 18:55:18,234 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=229 (was 337), ProcessCount=168 (was 173), AvailableMemoryMB=1260 (was 2459)