2023-02-13 10:01:45,925 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-02-13 10:01:46,183 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 10:01:46,273 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-13 10:01:46,321 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-13 10:01:46,327 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 10:01:46,337 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=338, ProcessCount=166, AvailableMemoryMB=7815 2023-02-13 10:01:46,339 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-13 10:01:46,339 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 10:01:46,462 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv100308763:84 , srv1017670282:84 , srv1022328065:84 , srv1026398435:84 , srv102803247:84 , srv1054975468:84 , srv1057489686:84 , srv1086153658:84 , srv1096996699:84 , srv1099455556:84 , srv1112116523:84 , srv1122918090:84 , srv1176528832:84 , srv117677867:84 , srv1191937586:84 , srv119317023:84 , srv1219240664:84 , srv1220061291:84 , srv1253035008:84 , srv126496248:84 , srv1265483492:84 , srv128842430:84 , srv1292085277:84 , srv1305110874:84 , srv1343531302:84 , srv1360563796:84 , srv1363092720:84 , srv1389126434:84 , srv1420061701:84 , srv1422264793:84 , srv143313675:84 , srv144661509:84 , srv1472055629:84 , srv1472172575:84 , srv1494921969:84 , srv152969329:84 , srv1535905822:84 , srv1541811904:84 , srv1548886522:84 , srv1555580543:84 , srv1570810931:84 , srv1608263359:84 , srv1629008398:84 , srv1647084792:84 , srv1684641312:84 , srv1717819960:84 , srv1728574142:84 , srv1762320466:84 , srv1782846715:84 , srv1791911660:84 , srv1820191757:84 , srv184777873:84 , srv1857205066:84 , srv1870240376:84 , srv1883169773:84 , srv1905291756:84 , srv1919052591:84 , srv1922283608:84 , srv1923077187:84 , srv1923156292:84 , srv1923326317:84 , srv1935640793:84 , srv1954381096:84 , srv1960581127:84 , srv2020504904:84 , srv208726591:84 , srv2121258657:84 , srv225785759:84 , srv260555352:84 , srv26509839:84 , srv269137145:84 , srv269165472:84 , srv273891747:84 , srv293366518:84 , srv34274126:84 , srv376846174:84 , srv419795342:84 , srv438755862:84 , srv476701329:84 , srv508582189:84 , srv511657621:84 , srv596140208:84 , srv596870335:84 , srv602465453:84 , srv607051644:84 , srv646781826:84 , srv690423293:84 , srv690603213:84 , srv69068976:84 , srv748307533:84 , srv758762246:84 , srv762391890:84 , srv77656085:84 , srv83215453:84 , srv838531441:84 , srv841189767:84 , srv875801054:84 , srv905593522:84 , srv923765927:84 , srv723006313:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-02-13 10:01:46,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 10:01:46,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1883169773=54, srv1923326317=60, srv1292085277=22, srv1360563796=25, srv117677867=13, srv1191937586=14, srv476701329=78, srv508582189=79, srv1422264793=29, srv1472172575=33, srv376846174=75, srv419795342=76, srv1112116523=10, srv511657621=80, srv1684641312=44, srv762391890=92, srv1935640793=61, srv748307533=90, srv100308763=0, srv1122918090=11, srv1022328065=2, srv143313675=30, srv838531441=95, srv438755862=77, srv273891747=72, srv596140208=81, srv1717819960=45, srv293366518=73, srv1791911660=49, srv1026398435=3, srv260555352=68, srv602465453=83, srv1923077187=58, srv1919052591=56, srv607051644=84, srv1954381096=62, srv905593522=98, srv1570810931=40, srv923765927=99, srv1548886522=38, srv2121258657=66, srv646781826=85, srv690603213=87, srv1265483492=20, srv1608263359=41, srv2020504904=64, srv1219240664=16, srv1629008398=42, srv1541811904=37, srv1054975468=5, srv1086153658=7, srv128842430=21, srv1728574142=46, srv1922283608=57, srv1057489686=6, srv596870335=82, srv69068976=88, srv1782846715=48, srv1017670282=1, srv102803247=4, srv1647084792=43, srv26509839=69, srv1905291756=55, srv1420061701=28, srv77656085=93, srv1363092720=26, srv119317023=15, srv208726591=65, srv269165472=71, srv1176528832=12, srv875801054=97, srv1472055629=32, srv758762246=91, srv269137145=70, srv126496248=19, srv1762320466=47, srv1253035008=18, srv144661509=31, srv1096996699=8, srv225785759=67, srv1099455556=9, srv34274126=74, srv1535905822=36, srv1857205066=52, srv1389126434=27, srv1960581127=63, srv83215453=94, srv1343531302=24, srv1870240376=53, srv1555580543=39, srv1494921969=34, srv1820191757=50, srv184777873=51, srv723006313=89, srv152969329=35, srv841189767=96, srv1220061291=17, srv690423293=86, srv1923156292=59, srv1305110874=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 10:01:46,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 10:01:46,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 10:01:46,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 10:01:46,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 10:01:46,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 10:01:46,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 10:01:46,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 10:01:46,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 10:01:46,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 10:01:46,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 10:01:46,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 10:01:46,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 10:01:46,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 10:01:46,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 10:01:46,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 10:01:46,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 10:01:46,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 10:01:46,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 10:01:46,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 10:01:46,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 10:01:46,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 10:01:46,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 10:01:46,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 10:01:46,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 10:01:46,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 10:01:46,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 10:01:46,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 10:01:46,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 10:01:46,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 10:01:46,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 10:01:46,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-13 10:01:46,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-13 10:01:46,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-13 10:01:46,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-13 10:01:46,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-13 10:01:46,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-13 10:01:46,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-13 10:01:46,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-13 10:01:46,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-13 10:01:46,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-13 10:01:46,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-13 10:01:46,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-13 10:01:46,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-13 10:01:46,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-13 10:01:46,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-13 10:01:46,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-13 10:01:46,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-13 10:01:46,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-13 10:01:46,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-13 10:01:46,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-13 10:01:46,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-13 10:01:46,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-13 10:01:46,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-13 10:01:46,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-13 10:01:46,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-13 10:01:46,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-13 10:01:46,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-13 10:01:46,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-13 10:01:46,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-13 10:01:46,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-13 10:01:46,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-13 10:01:46,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-13 10:01:46,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-13 10:01:46,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-13 10:01:46,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-13 10:01:46,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-13 10:01:46,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-13 10:01:46,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-13 10:01:46,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-13 10:01:46,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-13 10:01:46,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-13 10:01:46,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-13 10:01:46,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-13 10:01:46,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-13 10:01:46,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-13 10:01:46,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-13 10:01:46,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-13 10:01:46,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-13 10:01:46,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-13 10:01:46,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-13 10:01:46,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-13 10:01:46,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-13 10:01:46,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-13 10:01:46,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-13 10:01:46,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-13 10:01:46,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-13 10:01:46,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-13 10:01:46,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-13 10:01:46,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-13 10:01:46,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-13 10:01:46,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-13 10:01:46,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-13 10:01:46,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-13 10:01:46,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-13 10:01:46,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-13 10:01:46,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-13 10:01:46,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-13 10:01:46,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-13 10:01:46,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-13 10:01:46,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-13 10:01:46,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-13 10:01:46,837 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 10:01:46,838 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-13 10:01:46,838 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-13 10:01:47,163 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 563 ms to try 5000 different iterations. Found a solution that moves 4718 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17887033612997003. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03992729394281822, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05669933698743588, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5242222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03992729394281822, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23983333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 10:01:47,174 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv476701329:81 , srv1629008398:82 , srv607051644:82 , srv1026398435:83 , srv1057489686:83 , srv1253035008:83 , srv1265483492:83 , srv260555352:83 , srv905593522:83 , srv923765927:83 , srv100308763:84 , srv102803247:84 , srv1122918090:84 , srv119317023:84 , srv1684641312:84 , srv184777873:84 , srv1954381096:84 , srv269137145:84 , srv376846174:84 , srv438755862:84 , srv1054975468:85 , srv1086153658:85 , srv1096996699:85 , srv1176528832:85 , srv1191937586:85 , srv128842430:85 , srv1305110874:85 , srv1389126434:85 , srv144661509:85 , srv1535905822:85 , srv1548886522:85 , srv1820191757:85 , srv1870240376:85 , srv1922283608:85 , srv1923077187:85 , srv2121258657:85 , srv34274126:85 , srv508582189:85 , srv690423293:85 , srv762391890:85 , srv83215453:85 , srv1219240664:86 , srv1472055629:86 , srv152969329:86 , srv1570810931:86 , srv1608263359:86 , srv1935640793:86 , srv596140208:86 , srv596870335:86 , srv690603213:86 , srv758762246:86 , srv841189767:86 , srv126496248:87 , srv1363092720:87 , srv1422264793:87 , srv1472172575:87 , srv1555580543:87 , srv1857205066:87 , srv1960581127:87 , srv208726591:87 , srv225785759:87 , srv273891747:87 , srv293366518:87 , srv511657621:87 , srv838531441:87 , srv1022328065:88 , srv117677867:88 , srv1220061291:88 , srv1343531302:88 , srv1420061701:88 , srv1541811904:88 , srv1717819960:88 , srv1791911660:88 , srv1883169773:88 , srv1923156292:88 , srv1923326317:88 , srv419795342:88 , srv602465453:88 , srv69068976:88 , srv875801054:88 , srv1099455556:89 , srv1292085277:89 , srv143313675:89 , srv1647084792:89 , srv1905291756:89 , srv2020504904:89 , srv269165472:89 , srv646781826:89 , srv748307533:89 , srv77656085:89 , srv1017670282:90 , srv1112116523:90 , srv1360563796:90 , srv1762320466:90 , srv1494921969:91 , srv1728574142:91 , srv1782846715:91 , srv1919052591:91 , srv26509839:93 , srv723006313:445 } 2023-02-13 10:01:47,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 10:01:47,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1883169773=54, srv1923326317=60, srv1292085277=22, srv1360563796=25, srv117677867=13, srv1191937586=14, srv476701329=78, srv508582189=79, srv1422264793=29, srv1472172575=33, srv376846174=75, srv419795342=76, srv1112116523=10, srv511657621=80, srv1684641312=44, srv762391890=92, srv1935640793=61, srv748307533=90, srv100308763=0, srv1122918090=11, srv1022328065=2, srv143313675=30, srv838531441=95, srv438755862=77, srv273891747=72, srv596140208=81, srv1717819960=45, srv293366518=73, srv1791911660=49, srv1026398435=3, srv260555352=68, srv602465453=83, srv1923077187=58, srv1919052591=56, srv607051644=84, srv1954381096=62, srv905593522=98, srv1570810931=40, srv923765927=99, srv1548886522=38, srv2121258657=66, srv646781826=85, srv690603213=87, srv1265483492=20, srv1608263359=41, srv2020504904=64, srv1219240664=16, srv1629008398=42, srv1541811904=37, srv1054975468=5, srv1086153658=7, srv128842430=21, srv1728574142=46, srv1922283608=57, srv1057489686=6, srv596870335=82, srv69068976=88, srv1782846715=48, srv1017670282=1, srv102803247=4, srv1647084792=43, srv26509839=69, srv1905291756=55, srv1420061701=28, srv77656085=93, srv1363092720=26, srv119317023=15, srv208726591=65, srv269165472=71, srv1176528832=12, srv875801054=97, srv1472055629=32, srv758762246=91, srv269137145=70, srv126496248=19, srv1762320466=47, srv1253035008=18, srv144661509=31, srv1096996699=8, srv225785759=67, srv1099455556=9, srv34274126=74, srv1535905822=36, srv1857205066=52, srv1389126434=27, srv1960581127=63, srv83215453=94, srv1343531302=24, srv1870240376=53, srv1555580543=39, srv1494921969=34, srv1820191757=50, srv184777873=51, srv723006313=89, srv152969329=35, srv841189767=96, srv1220061291=17, srv690423293=86, srv1923156292=59, srv1305110874=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 10:01:47,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 10:01:47,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 10:01:47,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 10:01:47,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 10:01:47,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 10:01:47,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 10:01:47,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 10:01:47,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 10:01:47,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 10:01:47,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 10:01:47,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 10:01:47,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 10:01:47,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 10:01:47,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 10:01:47,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 10:01:47,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 10:01:47,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 10:01:47,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 10:01:47,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 10:01:47,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 10:01:47,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 10:01:47,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 10:01:47,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 10:01:47,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 10:01:47,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 10:01:47,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 10:01:47,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 10:01:47,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 10:01:47,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 10:01:47,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 10:01:47,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-13 10:01:47,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-13 10:01:47,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-13 10:01:47,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-13 10:01:47,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-13 10:01:47,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-13 10:01:47,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-13 10:01:47,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-13 10:01:47,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-13 10:01:47,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-13 10:01:47,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-13 10:01:47,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-13 10:01:47,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-13 10:01:47,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-13 10:01:47,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-13 10:01:47,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-13 10:01:47,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-13 10:01:47,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-13 10:01:47,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-13 10:01:47,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-13 10:01:47,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-13 10:01:47,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-13 10:01:47,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-13 10:01:47,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-13 10:01:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-13 10:01:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-13 10:01:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-13 10:01:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-13 10:01:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-13 10:01:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-13 10:01:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-13 10:01:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-13 10:01:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-13 10:01:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-13 10:01:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-13 10:01:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-13 10:01:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-13 10:01:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-13 10:01:47,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-13 10:01:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-13 10:01:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-13 10:01:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-13 10:01:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-13 10:01:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-13 10:01:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-13 10:01:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-13 10:01:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-13 10:01:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-13 10:01:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-13 10:01:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-13 10:01:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-13 10:01:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-13 10:01:47,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-13 10:01:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-13 10:01:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-13 10:01:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-13 10:01:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-13 10:01:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-13 10:01:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-13 10:01:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-13 10:01:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-13 10:01:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-13 10:01:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-13 10:01:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-13 10:01:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-13 10:01:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-13 10:01:47,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-13 10:01:47,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-13 10:01:47,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-13 10:01:47,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-13 10:01:47,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-13 10:01:47,298 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03992729394281822, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05669933698743588, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03992729394281822, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23983333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 10:01:47,298 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-13 10:01:47,298 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1788372999878744, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03992729394281822, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05669933698743588, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03992729394281822, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23983333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-13 10:01:47,487 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 273 ms to try 5000 different iterations. Found a solution that moves 2628 regions; Going from a computed imbalance of 0.1788372999878744 to a new imbalance of 0.01628512715948859. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022721660362024224); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036072629068328727, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.292, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022721660362024224); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01325); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04516666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 10:01:47,496 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv26509839:78 , srv1494921969:79 , srv1017670282:81 , srv1099455556:81 , srv1112116523:81 , srv1360563796:81 , srv1919052591:81 , srv748307533:81 , srv117677867:82 , srv1343531302:82 , srv143313675:82 , srv1728574142:82 , srv1782846715:82 , srv77656085:82 , srv1220061291:83 , srv1363092720:83 , srv1791911660:83 , srv1883169773:83 , srv1923326317:83 , srv1022328065:84 , srv1292085277:84 , srv1647084792:84 , srv1717819960:84 , srv1762320466:84 , srv1857205066:84 , srv1905291756:84 , srv273891747:84 , srv602465453:84 , srv838531441:84 , srv875801054:84 , srv1219240664:85 , srv126496248:85 , srv1420061701:85 , srv1422264793:85 , srv1535905822:85 , srv1541811904:85 , srv1820191757:85 , srv1923077187:85 , srv293366518:85 , srv419795342:85 , srv646781826:85 , srv69068976:85 , srv758762246:85 , srv762391890:85 , srv841189767:85 , srv1176528832:86 , srv1191937586:86 , srv128842430:86 , srv1389126434:86 , srv152969329:86 , srv1555580543:86 , srv1608263359:86 , srv1870240376:86 , srv1923156292:86 , srv1935640793:86 , srv2020504904:86 , srv208726591:86 , srv2121258657:86 , srv225785759:86 , srv269165472:86 , srv511657621:86 , srv596870335:86 , srv102803247:87 , srv1122918090:87 , srv119317023:87 , srv1253035008:87 , srv1305110874:87 , srv144661509:87 , srv1472055629:87 , srv1472172575:87 , srv1548886522:87 , srv1570810931:87 , srv1684641312:87 , srv184777873:87 , srv1922283608:87 , srv1960581127:87 , srv269137145:87 , srv34274126:87 , srv376846174:87 , srv905593522:87 , srv100308763:88 , srv1054975468:88 , srv1057489686:88 , srv508582189:88 , srv690423293:88 , srv690603213:88 , srv83215453:88 , srv1026398435:89 , srv1086153658:89 , srv1265483492:89 , srv1629008398:89 , srv476701329:89 , srv596140208:89 , srv923765927:89 , srv1096996699:90 , srv1954381096:90 , srv260555352:90 , srv438755862:90 , srv607051644:93 , srv723006313:531 } 2023-02-13 10:01:47,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 10:01:47,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1883169773=54, srv1923326317=60, srv1292085277=22, srv1360563796=25, srv117677867=13, srv1191937586=14, srv476701329=78, srv508582189=79, srv1422264793=29, srv1472172575=33, srv376846174=75, srv419795342=76, srv1112116523=10, srv511657621=80, srv1684641312=44, srv762391890=92, srv1935640793=61, srv748307533=90, srv100308763=0, srv1122918090=11, srv1022328065=2, srv143313675=30, srv838531441=95, srv438755862=77, srv273891747=72, srv596140208=81, srv1717819960=45, srv293366518=73, srv1791911660=49, srv1026398435=3, srv260555352=68, srv602465453=83, srv1923077187=58, srv1919052591=56, srv607051644=84, srv1954381096=62, srv905593522=98, srv1570810931=40, srv923765927=99, srv1548886522=38, srv2121258657=66, srv646781826=85, srv690603213=87, srv1265483492=20, srv1608263359=41, srv2020504904=64, srv1219240664=16, srv1629008398=42, srv1541811904=37, srv1054975468=5, srv1086153658=7, srv128842430=21, srv1728574142=46, srv1922283608=57, srv1057489686=6, srv596870335=82, srv69068976=88, srv1782846715=48, srv1017670282=1, srv102803247=4, srv1647084792=43, srv26509839=69, srv1905291756=55, srv1420061701=28, srv77656085=93, srv1363092720=26, srv119317023=15, srv208726591=65, srv269165472=71, srv1176528832=12, srv875801054=97, srv1472055629=32, srv758762246=91, srv269137145=70, srv126496248=19, srv1762320466=47, srv1253035008=18, srv144661509=31, srv1096996699=8, srv225785759=67, srv1099455556=9, srv34274126=74, srv1535905822=36, srv1857205066=52, srv1389126434=27, srv1960581127=63, srv83215453=94, srv1343531302=24, srv1870240376=53, srv1555580543=39, srv1494921969=34, srv1820191757=50, srv184777873=51, srv723006313=89, srv152969329=35, srv841189767=96, srv1220061291=17, srv690423293=86, srv1923156292=59, srv1305110874=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 10:01:47,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 10:01:47,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 10:01:47,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 10:01:47,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 10:01:47,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 10:01:47,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 10:01:47,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 10:01:47,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 10:01:47,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 10:01:47,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 10:01:47,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 10:01:47,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 10:01:47,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 10:01:47,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 10:01:47,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 10:01:47,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 10:01:47,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 10:01:47,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 10:01:47,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 10:01:47,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 10:01:47,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 10:01:47,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 10:01:47,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 10:01:47,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 10:01:47,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 10:01:47,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 10:01:47,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 10:01:47,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 10:01:47,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 10:01:47,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 10:01:47,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-13 10:01:47,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-13 10:01:47,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-13 10:01:47,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-13 10:01:47,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-13 10:01:47,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-13 10:01:47,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-13 10:01:47,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-13 10:01:47,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-13 10:01:47,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-13 10:01:47,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-13 10:01:47,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-13 10:01:47,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-13 10:01:47,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-13 10:01:47,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-13 10:01:47,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-13 10:01:47,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-13 10:01:47,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-13 10:01:47,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-13 10:01:47,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-13 10:01:47,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-13 10:01:47,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-13 10:01:47,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-13 10:01:47,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-13 10:01:47,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-13 10:01:47,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-13 10:01:47,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-13 10:01:47,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-13 10:01:47,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-13 10:01:47,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-13 10:01:47,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-13 10:01:47,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-13 10:01:47,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-13 10:01:47,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-13 10:01:47,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-13 10:01:47,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-13 10:01:47,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-13 10:01:47,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-13 10:01:47,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-13 10:01:47,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-13 10:01:47,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-13 10:01:47,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-13 10:01:47,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-13 10:01:47,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-13 10:01:47,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-13 10:01:47,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-13 10:01:47,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-13 10:01:47,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-13 10:01:47,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-13 10:01:47,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-13 10:01:47,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-13 10:01:47,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-13 10:01:47,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-13 10:01:47,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-13 10:01:47,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-13 10:01:47,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-13 10:01:47,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-13 10:01:47,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-13 10:01:47,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-13 10:01:47,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-13 10:01:47,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-13 10:01:47,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-13 10:01:47,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-13 10:01:47,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-13 10:01:47,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-13 10:01:47,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-13 10:01:47,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-13 10:01:47,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-13 10:01:47,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-13 10:01:47,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-13 10:01:47,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-13 10:01:47,604 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022721660362024224); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036072629068328727, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022721660362024224); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01325); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04516666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 10:01:47,604 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-13 10:01:47,605 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016266725510182253, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022721660362024224); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.036072629068328727, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022721660362024224); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01325); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04516666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-13 10:01:47,763 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 232 ms to try 5000 different iterations. Found a solution that moves 1024 regions; Going from a computed imbalance of 0.016266725510182253 to a new imbalance of 6.574677819508917E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0057616108096681645); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008301285176284588); MoveCostFunction : (multiplier=7.0, imbalance=0.11377777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0057616108096681645); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 10:01:47,765 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv596140208:82 , srv1096996699:83 , srv1472172575:83 , srv1762320466:83 , srv2020504904:83 , srv269165472:83 , srv438755862:83 , srv607051644:83 , srv646781826:83 , srv1054975468:84 , srv1086153658:84 , srv1420061701:84 , srv1647084792:84 , srv1905291756:84 , srv1923156292:84 , srv1954381096:84 , srv225785759:84 , srv260555352:84 , srv419795342:84 , srv690603213:84 , srv69068976:84 , srv83215453:84 , srv100308763:85 , srv1022328065:85 , srv1220061291:85 , srv1292085277:85 , srv1305110874:85 , srv144661509:85 , srv1472055629:85 , srv1541811904:85 , srv1555580543:85 , srv1570810931:85 , srv1608263359:85 , srv1728574142:85 , srv1782846715:85 , srv1919052591:85 , srv1922283608:85 , srv1935640793:85 , srv1960581127:85 , srv208726591:85 , srv34274126:85 , srv508582189:85 , srv511657621:85 , srv596870335:85 , srv875801054:85 , srv1026398435:86 , srv102803247:86 , srv1112116523:86 , srv1176528832:86 , srv119317023:86 , srv1219240664:86 , srv126496248:86 , srv1265483492:86 , srv128842430:86 , srv1360563796:86 , srv1422264793:86 , srv152969329:86 , srv1548886522:86 , srv184777873:86 , srv1870240376:86 , srv1883169773:86 , srv1923326317:86 , srv2121258657:86 , srv269137145:86 , srv273891747:86 , srv293366518:86 , srv376846174:86 , srv690423293:86 , srv758762246:86 , srv77656085:86 , srv841189767:86 , srv923765927:86 , srv1017670282:87 , srv1057489686:87 , srv1122918090:87 , srv117677867:87 , srv1191937586:87 , srv1253035008:87 , srv1343531302:87 , srv1363092720:87 , srv1389126434:87 , srv143313675:87 , srv1494921969:87 , srv1629008398:87 , srv1684641312:87 , srv1717819960:87 , srv1791911660:87 , srv1857205066:87 , srv1923077187:87 , srv26509839:87 , srv476701329:87 , srv602465453:87 , srv748307533:87 , srv762391890:87 , srv838531441:87 , srv905593522:87 , srv1099455556:88 , srv1535905822:88 , srv1820191757:88 , srv723006313:533 } 2023-02-13 10:01:47,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 10:01:47,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1883169773=54, srv1923326317=60, srv1292085277=22, srv1360563796=25, srv117677867=13, srv1191937586=14, srv476701329=78, srv508582189=79, srv1422264793=29, srv1472172575=33, srv376846174=75, srv419795342=76, srv1112116523=10, srv511657621=80, srv1684641312=44, srv762391890=92, srv1935640793=61, srv748307533=90, srv100308763=0, srv1122918090=11, srv1022328065=2, srv143313675=30, srv838531441=95, srv438755862=77, srv273891747=72, srv596140208=81, srv1717819960=45, srv293366518=73, srv1791911660=49, srv1026398435=3, srv260555352=68, srv602465453=83, srv1923077187=58, srv1919052591=56, srv607051644=84, srv1954381096=62, srv905593522=98, srv1570810931=40, srv923765927=99, srv1548886522=38, srv2121258657=66, srv646781826=85, srv690603213=87, srv1265483492=20, srv1608263359=41, srv2020504904=64, srv1219240664=16, srv1629008398=42, srv1541811904=37, srv1054975468=5, srv1086153658=7, srv128842430=21, srv1728574142=46, srv1922283608=57, srv1057489686=6, srv596870335=82, srv69068976=88, srv1782846715=48, srv1017670282=1, srv102803247=4, srv1647084792=43, srv26509839=69, srv1905291756=55, srv1420061701=28, srv77656085=93, srv1363092720=26, srv119317023=15, srv208726591=65, srv269165472=71, srv1176528832=12, srv875801054=97, srv1472055629=32, srv758762246=91, srv269137145=70, srv126496248=19, srv1762320466=47, srv1253035008=18, srv144661509=31, srv1096996699=8, srv225785759=67, srv1099455556=9, srv34274126=74, srv1535905822=36, srv1857205066=52, srv1389126434=27, srv1960581127=63, srv83215453=94, srv1343531302=24, srv1870240376=53, srv1555580543=39, srv1494921969=34, srv1820191757=50, srv184777873=51, srv723006313=89, srv152969329=35, srv841189767=96, srv1220061291=17, srv690423293=86, srv1923156292=59, srv1305110874=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 10:01:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 10:01:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 10:01:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 10:01:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 10:01:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 10:01:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 10:01:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 10:01:47,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 10:01:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 10:01:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 10:01:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 10:01:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 10:01:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 10:01:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 10:01:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 10:01:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 10:01:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 10:01:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 10:01:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 10:01:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 10:01:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 10:01:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 10:01:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 10:01:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 10:01:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 10:01:47,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 10:01:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 10:01:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 10:01:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 10:01:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 10:01:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-13 10:01:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-13 10:01:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-13 10:01:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-13 10:01:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-13 10:01:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-13 10:01:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-13 10:01:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-13 10:01:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-13 10:01:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-13 10:01:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-13 10:01:47,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-13 10:01:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-13 10:01:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-13 10:01:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-13 10:01:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-13 10:01:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-13 10:01:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-13 10:01:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-13 10:01:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-13 10:01:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-13 10:01:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-13 10:01:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-13 10:01:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-13 10:01:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-13 10:01:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-13 10:01:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-13 10:01:47,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-13 10:01:47,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-13 10:01:47,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-13 10:01:47,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-13 10:01:47,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-13 10:01:47,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-13 10:01:47,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-13 10:01:47,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-13 10:01:47,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-13 10:01:47,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-13 10:01:47,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-13 10:01:47,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-13 10:01:47,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-13 10:01:47,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-13 10:01:47,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-13 10:01:47,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-13 10:01:47,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-13 10:01:47,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-13 10:01:47,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-13 10:01:47,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-13 10:01:47,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-13 10:01:47,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-13 10:01:47,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-13 10:01:47,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-13 10:01:47,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-13 10:01:47,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-13 10:01:47,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-13 10:01:47,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-13 10:01:47,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-13 10:01:47,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-13 10:01:47,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-13 10:01:47,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-13 10:01:47,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-13 10:01:47,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-13 10:01:47,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-13 10:01:47,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-13 10:01:47,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-13 10:01:47,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-13 10:01:47,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-13 10:01:47,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-13 10:01:47,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-13 10:01:47,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-13 10:01:47,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-13 10:01:47,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-13 10:01:47,861 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0057616108096681645); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008301285176284588); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0057616108096681645); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 10:01:47,862 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-13 10:01:47,862 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.502975806991075E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0057616108096681645); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008301285176284588); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0057616108096681645); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-13 10:01:48,074 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 284 ms to try 5000 different iterations. Found a solution that moves 263 regions; Going from a computed imbalance of 6.502975806991075E-4 to a new imbalance of 4.479603586579635E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023688968483956715); MoveCostFunction : (multiplier=7.0, imbalance=0.029222222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 10:01:48,075 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1099455556:83 , srv1191937586:83 , srv1717819960:83 , srv1791911660:83 , srv1857205066:83 , srv1960581127:83 , srv208726591:83 , srv602465453:83 , srv690423293:83 , srv1017670282:84 , srv1026398435:84 , srv1054975468:84 , srv1057489686:84 , srv1086153658:84 , srv126496248:84 , srv1265483492:84 , srv1292085277:84 , srv1422264793:84 , srv143313675:84 , srv1472055629:84 , srv1472172575:84 , srv152969329:84 , srv1535905822:84 , srv1541811904:84 , srv1548886522:84 , srv1555580543:84 , srv1570810931:84 , srv1629008398:84 , srv1684641312:84 , srv1728574142:84 , srv1782846715:84 , srv1820191757:84 , srv1922283608:84 , srv1923156292:84 , srv1954381096:84 , srv2020504904:84 , srv26509839:84 , srv269165472:84 , srv293366518:84 , srv508582189:84 , srv511657621:84 , srv607051644:84 , srv690603213:84 , srv77656085:84 , srv838531441:84 , srv100308763:85 , srv1022328065:85 , srv102803247:85 , srv1096996699:85 , srv1112116523:85 , srv1122918090:85 , srv1176528832:85 , srv117677867:85 , srv119317023:85 , srv1219240664:85 , srv1253035008:85 , srv1305110874:85 , srv1343531302:85 , srv1360563796:85 , srv1363092720:85 , srv1389126434:85 , srv1420061701:85 , srv144661509:85 , srv1608263359:85 , srv1647084792:85 , srv1762320466:85 , srv184777873:85 , srv1870240376:85 , srv1883169773:85 , srv1905291756:85 , srv1919052591:85 , srv1923077187:85 , srv1923326317:85 , srv1935640793:85 , srv2121258657:85 , srv225785759:85 , srv260555352:85 , srv269137145:85 , srv273891747:85 , srv34274126:85 , srv376846174:85 , srv419795342:85 , srv438755862:85 , srv476701329:85 , srv596140208:85 , srv646781826:85 , srv69068976:85 , srv748307533:85 , srv758762246:85 , srv762391890:85 , srv83215453:85 , srv875801054:85 , srv905593522:85 , srv923765927:85 , srv1220061291:86 , srv128842430:86 , srv1494921969:86 , srv596870335:86 , srv841189767:86 , srv723006313:634 } 2023-02-13 10:01:48,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 10:01:48,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1883169773=54, srv1923326317=60, srv1292085277=22, srv1360563796=25, srv117677867=13, srv1191937586=14, srv476701329=78, srv508582189=79, srv1422264793=29, srv1472172575=33, srv376846174=75, srv419795342=76, srv1112116523=10, srv511657621=80, srv1684641312=44, srv762391890=92, srv1935640793=61, srv748307533=90, srv100308763=0, srv1122918090=11, srv1022328065=2, srv143313675=30, srv838531441=95, srv438755862=77, srv273891747=72, srv596140208=81, srv1717819960=45, srv293366518=73, srv1791911660=49, srv1026398435=3, srv260555352=68, srv602465453=83, srv1923077187=58, srv1919052591=56, srv607051644=84, srv1954381096=62, srv905593522=98, srv1570810931=40, srv923765927=99, srv1548886522=38, srv2121258657=66, srv646781826=85, srv690603213=87, srv1265483492=20, srv1608263359=41, srv2020504904=64, srv1219240664=16, srv1629008398=42, srv1541811904=37, srv1054975468=5, srv1086153658=7, srv128842430=21, srv1728574142=46, srv1922283608=57, srv1057489686=6, srv596870335=82, srv69068976=88, srv1782846715=48, srv1017670282=1, srv102803247=4, srv1647084792=43, srv26509839=69, srv1905291756=55, srv1420061701=28, srv77656085=93, srv1363092720=26, srv119317023=15, srv208726591=65, srv269165472=71, srv1176528832=12, srv875801054=97, srv1472055629=32, srv758762246=91, srv269137145=70, srv126496248=19, srv1762320466=47, srv1253035008=18, srv144661509=31, srv1096996699=8, srv225785759=67, srv1099455556=9, srv34274126=74, srv1535905822=36, srv1857205066=52, srv1389126434=27, srv1960581127=63, srv83215453=94, srv1343531302=24, srv1870240376=53, srv1555580543=39, srv1494921969=34, srv1820191757=50, srv184777873=51, srv723006313=89, srv152969329=35, srv841189767=96, srv1220061291=17, srv690423293=86, srv1923156292=59, srv1305110874=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 10:01:48,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 10:01:48,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 10:01:48,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 10:01:48,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 10:01:48,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 10:01:48,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 10:01:48,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 10:01:48,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 10:01:48,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 10:01:48,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 10:01:48,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 10:01:48,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 10:01:48,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 10:01:48,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 10:01:48,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 10:01:48,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 10:01:48,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 10:01:48,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 10:01:48,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 10:01:48,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 10:01:48,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 10:01:48,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 10:01:48,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 10:01:48,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 10:01:48,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 10:01:48,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 10:01:48,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 10:01:48,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 10:01:48,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 10:01:48,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 10:01:48,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-13 10:01:48,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-13 10:01:48,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-13 10:01:48,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-13 10:01:48,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-13 10:01:48,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-13 10:01:48,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-13 10:01:48,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-13 10:01:48,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-13 10:01:48,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-13 10:01:48,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-13 10:01:48,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-13 10:01:48,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-13 10:01:48,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-13 10:01:48,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-13 10:01:48,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-13 10:01:48,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-13 10:01:48,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-13 10:01:48,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-13 10:01:48,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-13 10:01:48,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-13 10:01:48,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-13 10:01:48,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-13 10:01:48,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-13 10:01:48,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-13 10:01:48,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-13 10:01:48,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-13 10:01:48,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-13 10:01:48,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-13 10:01:48,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-13 10:01:48,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-13 10:01:48,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-13 10:01:48,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-13 10:01:48,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-13 10:01:48,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-13 10:01:48,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-13 10:01:48,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-13 10:01:48,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-13 10:01:48,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-13 10:01:48,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-13 10:01:48,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-13 10:01:48,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-13 10:01:48,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-13 10:01:48,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-13 10:01:48,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-13 10:01:48,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-13 10:01:48,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-13 10:01:48,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-13 10:01:48,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-13 10:01:48,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-13 10:01:48,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-13 10:01:48,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-13 10:01:48,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-13 10:01:48,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-13 10:01:48,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-13 10:01:48,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-13 10:01:48,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-13 10:01:48,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-13 10:01:48,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-13 10:01:48,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-13 10:01:48,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-13 10:01:48,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-13 10:01:48,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-13 10:01:48,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-13 10:01:48,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-13 10:01:48,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-13 10:01:48,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-13 10:01:48,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-13 10:01:48,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-13 10:01:48,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-13 10:01:48,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-13 10:01:48,175 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023688968483956715); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 10:01:48,175 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-13 10:01:48,175 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.295447050523066E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023688968483956715); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-13 10:01:48,318 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 217 ms to try 5000 different iterations. Found a solution that moves 36 regions; Going from a computed imbalance of 4.295447050523066E-5 to a new imbalance of 7.639834888070601E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.004); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 10:01:48,318 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1096996699:83 , srv1122918090:83 , srv128842430:83 , srv1389126434:83 , srv1494921969:83 , srv596870335:83 , srv841189767:83 , srv923765927:83 , srv100308763:84 , srv1017670282:84 , srv1022328065:84 , srv1026398435:84 , srv102803247:84 , srv1057489686:84 , srv1086153658:84 , srv1112116523:84 , srv1176528832:84 , srv117677867:84 , srv119317023:84 , srv1219240664:84 , srv1220061291:84 , srv1253035008:84 , srv126496248:84 , srv1265483492:84 , srv1292085277:84 , srv1305110874:84 , srv1343531302:84 , srv1360563796:84 , srv1363092720:84 , srv1420061701:84 , srv1422264793:84 , srv143313675:84 , srv144661509:84 , srv1472055629:84 , srv152969329:84 , srv1535905822:84 , srv1541811904:84 , srv1548886522:84 , srv1555580543:84 , srv1570810931:84 , srv1608263359:84 , srv1629008398:84 , srv1647084792:84 , srv1684641312:84 , srv1717819960:84 , srv1728574142:84 , srv1762320466:84 , srv1782846715:84 , srv1820191757:84 , srv184777873:84 , srv1870240376:84 , srv1883169773:84 , srv1905291756:84 , srv1919052591:84 , srv1923077187:84 , srv1923156292:84 , srv1923326317:84 , srv1935640793:84 , srv1954381096:84 , srv1960581127:84 , srv2020504904:84 , srv2121258657:84 , srv225785759:84 , srv260555352:84 , srv26509839:84 , srv269137145:84 , srv269165472:84 , srv273891747:84 , srv293366518:84 , srv34274126:84 , srv376846174:84 , srv419795342:84 , srv438755862:84 , srv476701329:84 , srv508582189:84 , srv511657621:84 , srv596140208:84 , srv602465453:84 , srv607051644:84 , srv646781826:84 , srv690423293:84 , srv690603213:84 , srv69068976:84 , srv748307533:84 , srv758762246:84 , srv762391890:84 , srv83215453:84 , srv838531441:84 , srv875801054:84 , srv905593522:84 , srv1054975468:85 , srv1099455556:85 , srv1191937586:85 , srv1472172575:85 , srv1791911660:85 , srv1857205066:85 , srv1922283608:85 , srv208726591:85 , srv77656085:85 , srv723006313:683 } 2023-02-13 10:01:48,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 10:01:48,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1883169773=54, srv1923326317=60, srv1292085277=22, srv1360563796=25, srv117677867=13, srv1191937586=14, srv476701329=78, srv508582189=79, srv1422264793=29, srv1472172575=33, srv376846174=75, srv419795342=76, srv1112116523=10, srv511657621=80, srv1684641312=44, srv762391890=92, srv1935640793=61, srv748307533=90, srv100308763=0, srv1122918090=11, srv1022328065=2, srv143313675=30, srv838531441=95, srv438755862=77, srv273891747=72, srv596140208=81, srv1717819960=45, srv293366518=73, srv1791911660=49, srv1026398435=3, srv260555352=68, srv602465453=83, srv1923077187=58, srv1919052591=56, srv607051644=84, srv1954381096=62, srv905593522=98, srv1570810931=40, srv923765927=99, srv1548886522=38, srv2121258657=66, srv646781826=85, srv690603213=87, srv1265483492=20, srv1608263359=41, srv2020504904=64, srv1219240664=16, srv1629008398=42, srv1541811904=37, srv1054975468=5, srv1086153658=7, srv128842430=21, srv1728574142=46, srv1922283608=57, srv1057489686=6, srv596870335=82, srv69068976=88, srv1782846715=48, srv1017670282=1, srv102803247=4, srv1647084792=43, srv26509839=69, srv1905291756=55, srv1420061701=28, srv77656085=93, srv1363092720=26, srv119317023=15, srv208726591=65, srv269165472=71, srv1176528832=12, srv875801054=97, srv1472055629=32, srv758762246=91, srv269137145=70, srv126496248=19, srv1762320466=47, srv1253035008=18, srv144661509=31, srv1096996699=8, srv225785759=67, srv1099455556=9, srv34274126=74, srv1535905822=36, srv1857205066=52, srv1389126434=27, srv1960581127=63, srv83215453=94, srv1343531302=24, srv1870240376=53, srv1555580543=39, srv1494921969=34, srv1820191757=50, srv184777873=51, srv723006313=89, srv152969329=35, srv841189767=96, srv1220061291=17, srv690423293=86, srv1923156292=59, srv1305110874=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 10:01:48,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 10:01:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 10:01:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 10:01:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 10:01:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 10:01:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 10:01:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 10:01:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 10:01:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 10:01:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 10:01:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 10:01:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 10:01:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 10:01:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 10:01:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 10:01:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 10:01:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 10:01:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 10:01:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 10:01:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 10:01:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 10:01:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 10:01:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 10:01:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 10:01:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 10:01:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 10:01:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 10:01:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 10:01:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 10:01:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 10:01:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-13 10:01:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-13 10:01:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-13 10:01:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-13 10:01:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-13 10:01:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-13 10:01:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-13 10:01:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-13 10:01:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-13 10:01:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-13 10:01:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-13 10:01:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-13 10:01:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-13 10:01:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-13 10:01:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-13 10:01:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-13 10:01:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-13 10:01:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-13 10:01:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-13 10:01:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-13 10:01:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-13 10:01:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-13 10:01:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-13 10:01:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-13 10:01:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-13 10:01:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-13 10:01:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-13 10:01:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-13 10:01:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-13 10:01:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-13 10:01:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-13 10:01:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-13 10:01:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-13 10:01:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-13 10:01:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-13 10:01:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-13 10:01:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-13 10:01:48,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-13 10:01:48,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-13 10:01:48,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-13 10:01:48,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-13 10:01:48,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-13 10:01:48,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-13 10:01:48,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-13 10:01:48,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-13 10:01:48,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-13 10:01:48,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-13 10:01:48,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-13 10:01:48,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-13 10:01:48,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-13 10:01:48,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-13 10:01:48,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-13 10:01:48,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-13 10:01:48,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-13 10:01:48,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-13 10:01:48,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-13 10:01:48,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-13 10:01:48,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-13 10:01:48,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-13 10:01:48,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-13 10:01:48,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-13 10:01:48,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-13 10:01:48,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-13 10:01:48,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-13 10:01:48,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-13 10:01:48,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-13 10:01:48,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-13 10:01:48,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-13 10:01:48,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-13 10:01:48,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-13 10:01:48,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-13 10:01:48,420 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 10:01:48,420 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1096996699:83 , srv1122918090:83 , srv128842430:83 , srv1389126434:83 , srv1494921969:83 , srv596870335:83 , srv841189767:83 , srv923765927:83 , srv100308763:84 , srv1017670282:84 , srv1022328065:84 , srv1026398435:84 , srv102803247:84 , srv1057489686:84 , srv1086153658:84 , srv1112116523:84 , srv1176528832:84 , srv117677867:84 , srv119317023:84 , srv1219240664:84 , srv1220061291:84 , srv1253035008:84 , srv126496248:84 , srv1265483492:84 , srv1292085277:84 , srv1305110874:84 , srv1343531302:84 , srv1360563796:84 , srv1363092720:84 , srv1420061701:84 , srv1422264793:84 , srv143313675:84 , srv144661509:84 , srv1472055629:84 , srv152969329:84 , srv1535905822:84 , srv1541811904:84 , srv1548886522:84 , srv1555580543:84 , srv1570810931:84 , srv1608263359:84 , srv1629008398:84 , srv1647084792:84 , srv1684641312:84 , srv1717819960:84 , srv1728574142:84 , srv1762320466:84 , srv1782846715:84 , srv1820191757:84 , srv184777873:84 , srv1870240376:84 , srv1883169773:84 , srv1905291756:84 , srv1919052591:84 , srv1923077187:84 , srv1923156292:84 , srv1923326317:84 , srv1935640793:84 , srv1954381096:84 , srv1960581127:84 , srv2020504904:84 , srv2121258657:84 , srv225785759:84 , srv260555352:84 , srv26509839:84 , srv269137145:84 , srv269165472:84 , srv273891747:84 , srv293366518:84 , srv34274126:84 , srv376846174:84 , srv419795342:84 , srv438755862:84 , srv476701329:84 , srv508582189:84 , srv511657621:84 , srv596140208:84 , srv602465453:84 , srv607051644:84 , srv646781826:84 , srv690423293:84 , srv690603213:84 , srv69068976:84 , srv748307533:84 , srv758762246:84 , srv762391890:84 , srv83215453:84 , srv838531441:84 , srv875801054:84 , srv905593522:84 , srv1054975468:85 , srv1099455556:85 , srv1191937586:85 , srv1472172575:85 , srv1791911660:85 , srv1857205066:85 , srv1922283608:85 , srv208726591:85 , srv77656085:85 , srv723006313:683 } 2023-02-13 10:01:48,441 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=327 (was 338), ProcessCount=166 (was 166), AvailableMemoryMB=7322 (was 7815) 2023-02-13 10:01:48,448 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=327, ProcessCount=166, AvailableMemoryMB=7322 2023-02-13 10:01:48,448 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-13 10:01:48,448 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-02-13 10:01:48,458 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1018560562:84 , srv116035534:84 , srv1190929186:84 , srv1274074671:84 , srv1361814417:84 , srv140302658:84 , srv1499334909:84 , srv1505166144:84 , srv1657781281:84 , srv16877374:84 , srv1738618804:84 , srv1745320933:84 , srv1771440351:84 , srv1875370414:84 , srv1976690361:84 , srv2037052346:84 , srv207161132:84 , srv2075907766:84 , srv2101053009:84 , srv213913349:84 , srv238427400:84 , srv249861178:84 , srv280869232:84 , srv334132408:84 , srv339031749:84 , srv38338246:84 , srv613177003:84 , srv771432443:84 , srv946672400:84 , srv2009948200:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-02-13 10:01:48,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 10:01:48,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1657781281=8, srv16877374=9, srv1976690361=14, srv1499334909=6, srv1745320933=11, srv1361814417=4, srv1738618804=10, srv116035534=1, srv1771440351=12, srv2009948200=15, srv2075907766=18, srv213913349=20, srv140302658=5, srv38338246=26, srv1875370414=13, srv1018560562=0, srv1274074671=3, srv249861178=22, srv280869232=23, srv613177003=27, srv1190929186=2, srv2101053009=19, srv946672400=29, srv1505166144=7, srv2037052346=16, srv334132408=24, srv238427400=21, srv771432443=28, srv339031749=25, srv207161132=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 10:01:48,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 10:01:48,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 10:01:48,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 10:01:48,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 10:01:48,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 10:01:48,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 10:01:48,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 10:01:48,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 10:01:48,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 10:01:48,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 10:01:48,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 10:01:48,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 10:01:48,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 10:01:48,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 10:01:48,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 10:01:48,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 10:01:48,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 10:01:48,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 10:01:48,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 10:01:48,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 10:01:48,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 10:01:48,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 10:01:48,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 10:01:48,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 10:01:48,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 10:01:48,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 10:01:48,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 10:01:48,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 10:01:48,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 10:01:48,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 10:01:48,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-13 10:01:48,502 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1066047949872477, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 10:01:48,502 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912528951492318, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1066047949872477, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-02-13 10:03:48,481 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9747515 different iterations. Found a solution that moves 1837 regions; Going from a computed imbalance of 0.9912528951492318 to a new imbalance of 4.2876496417733574E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6803703703703704, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 10:03:48,486 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1018560562:90 , srv116035534:90 , srv1190929186:90 , srv1274074671:90 , srv1361814417:90 , srv140302658:90 , srv1499334909:90 , srv1505166144:90 , srv1657781281:90 , srv16877374:90 , srv1738618804:90 , srv1745320933:90 , srv1771440351:90 , srv1875370414:90 , srv1976690361:90 , srv2009948200:90 , srv2037052346:90 , srv207161132:90 , srv2075907766:90 , srv2101053009:90 , srv213913349:90 , srv238427400:90 , srv249861178:90 , srv280869232:90 , srv334132408:90 , srv339031749:90 , srv38338246:90 , srv613177003:90 , srv771432443:90 , srv946672400:90 } 2023-02-13 10:03:48,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-13 10:03:48,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1657781281=8, srv16877374=9, srv1976690361=14, srv1499334909=6, srv1745320933=11, srv1361814417=4, srv1738618804=10, srv116035534=1, srv1771440351=12, srv2009948200=15, srv2075907766=18, srv213913349=20, srv140302658=5, srv38338246=26, srv1875370414=13, srv1018560562=0, srv1274074671=3, srv249861178=22, srv280869232=23, srv613177003=27, srv1190929186=2, srv2101053009=19, srv946672400=29, srv1505166144=7, srv2037052346=16, srv334132408=24, srv238427400=21, srv771432443=28, srv339031749=25, srv207161132=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-13 10:03:48,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 10:03:48,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-13 10:03:48,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-13 10:03:48,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-13 10:03:48,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-13 10:03:48,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-13 10:03:48,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-13 10:03:48,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-13 10:03:48,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-13 10:03:48,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-13 10:03:48,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-13 10:03:48,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-13 10:03:48,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-13 10:03:48,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-13 10:03:48,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-13 10:03:48,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-13 10:03:48,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-13 10:03:48,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-13 10:03:48,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-13 10:03:48,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-13 10:03:48,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-13 10:03:48,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-13 10:03:48,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-13 10:03:48,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-13 10:03:48,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-13 10:03:48,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-13 10:03:48,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-13 10:03:48,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-13 10:03:48,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-13 10:03:48,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-13 10:03:48,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-13 10:03:48,501 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-13 10:03:48,501 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1018560562:90 , srv116035534:90 , srv1190929186:90 , srv1274074671:90 , srv1361814417:90 , srv140302658:90 , srv1499334909:90 , srv1505166144:90 , srv1657781281:90 , srv16877374:90 , srv1738618804:90 , srv1745320933:90 , srv1771440351:90 , srv1875370414:90 , srv1976690361:90 , srv2009948200:90 , srv2037052346:90 , srv207161132:90 , srv2075907766:90 , srv2101053009:90 , srv213913349:90 , srv238427400:90 , srv249861178:90 , srv280869232:90 , srv334132408:90 , srv339031749:90 , srv38338246:90 , srv613177003:90 , srv771432443:90 , srv946672400:90 } 2023-02-13 10:03:48,512 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=192 (was 327), ProcessCount=166 (was 166), AvailableMemoryMB=7153 (was 7322)