2023-08-15 11:53:09,568 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-08-15 11:53:09,900 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-08-15 11:53:10,011 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-08-15 11:53:10,087 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-15 11:53:10,096 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-15 11:53:10,112 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=264, ProcessCount=171, AvailableMemoryMB=3127 2023-08-15 11:53:10,115 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-15 11:53:10,115 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-15 11:53:10,243 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1080505292:84 , srv1096541632:84 , srv1126309055:84 , srv113462942:84 , srv1168839644:84 , srv1182566185:84 , srv1210555252:84 , srv123010808:84 , srv1255774301:84 , srv1259491330:84 , srv1263971624:84 , srv126970828:84 , srv1277373608:84 , srv1292360648:84 , srv1297686668:84 , srv1314238935:84 , srv1325342040:84 , srv1338248177:84 , srv1457370374:84 , srv1474549858:84 , srv1474832259:84 , srv1501393740:84 , srv1514079114:84 , srv1522329372:84 , srv1526683479:84 , srv1527971759:84 , srv1571047453:84 , srv1575028851:84 , srv1583983527:84 , srv1679041824:84 , srv1708699763:84 , srv1736293470:84 , srv1753002889:84 , srv1754954443:84 , srv1763569383:84 , srv1770459162:84 , srv1773437254:84 , srv1792004007:84 , srv1803488849:84 , srv1822198475:84 , srv182938569:84 , srv1864672934:84 , srv1908770240:84 , srv1946457003:84 , srv1992945499:84 , srv2036591050:84 , srv2039224540:84 , srv2060225161:84 , srv2073952959:84 , srv2081676281:84 , srv2082418442:84 , srv2087672772:84 , srv2097571299:84 , srv2137138743:84 , srv238710898:84 , srv24288931:84 , srv273168840:84 , srv290517432:84 , srv310979103:84 , srv328640100:84 , srv330195515:84 , srv335475499:84 , srv368125431:84 , srv396784793:84 , srv400744726:84 , srv431626909:84 , srv482173621:84 , srv493457165:84 , srv523472923:84 , srv547231258:84 , srv611426675:84 , srv612249308:84 , srv617160168:84 , srv656386454:84 , srv660559289:84 , srv696545221:84 , srv721541665:84 , srv724179225:84 , srv730521589:84 , srv737957241:84 , srv755062870:84 , srv765865784:84 , srv79750131:84 , srv79986197:84 , srv814463233:84 , srv825219335:84 , srv839511415:84 , srv848979275:84 , srv850901261:84 , srv873482528:84 , srv894096847:84 , srv902196171:84 , srv903976379:84 , srv921051127:84 , srv931320642:84 , srv945804813:84 , srv96856434:84 , srv981942602:84 , srv990548794:84 , srv1411080676:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-08-15 11:53:10,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-15 11:53:10,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2097571299=53, srv79986197=84, srv990548794=99, srv1753002889=33, srv1314238935=15, srv1679041824=30, srv123010808=7, srv755062870=81, srv1763569383=35, srv2137138743=54, srv1263971624=10, srv1571047453=27, srv1754954443=34, srv400744726=65, srv1526683479=25, srv1708699763=31, srv113462942=3, srv24288931=56, srv2060225161=48, srv1474832259=21, srv1527971759=26, srv612249308=72, srv79750131=83, srv96856434=97, srv1864672934=42, srv2082418442=51, srv523472923=69, srv1770459162=36, srv1575028851=28, srv1080505292=0, srv696545221=76, srv1803488849=39, srv721541665=77, srv1946457003=44, srv765865784=82, srv368125431=63, srv1096541632=1, srv1338248177=17, srv126970828=11, srv1992945499=45, srv903976379=93, srv1210555252=6, srv1736293470=32, srv1292360648=13, srv902196171=92, srv931320642=95, srv396784793=64, srv1411080676=18, srv238710898=55, srv981942602=98, srv1474549858=20, srv730521589=79, srv1908770240=43, srv2039224540=47, srv1514079114=23, srv335475499=62, srv1297686668=14, srv1255774301=8, srv2073952959=49, srv1277373608=12, srv2081676281=50, srv848979275=88, srv1457370374=19, srv2036591050=46, srv656386454=74, srv431626909=66, srv724179225=78, srv1822198475=40, srv1522329372=24, srv330195515=61, srv814463233=85, srv1259491330=9, srv1182566185=5, srv1168839644=4, srv547231258=70, srv1126309055=2, srv921051127=94, srv1325342040=16, srv482173621=67, srv1501393740=22, srv617160168=73, srv894096847=91, srv182938569=41, srv273168840=57, srv873482528=90, srv660559289=75, srv1792004007=38, srv2087672772=52, srv825219335=86, srv290517432=58, srv945804813=96, srv850901261=89, srv1583983527=29, srv328640100=60, srv611426675=71, srv1773437254=37, srv737957241=80, srv493457165=68, srv839511415=87, srv310979103=59} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-15 11:53:10,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-15 11:53:10,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-15 11:53:10,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-15 11:53:10,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-15 11:53:10,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-15 11:53:10,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-15 11:53:10,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-15 11:53:10,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-15 11:53:10,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-15 11:53:10,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-15 11:53:10,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-15 11:53:10,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-15 11:53:10,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-15 11:53:10,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-15 11:53:10,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-15 11:53:10,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-15 11:53:10,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-15 11:53:10,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-15 11:53:10,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-15 11:53:10,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-15 11:53:10,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-15 11:53:10,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-15 11:53:10,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-15 11:53:10,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-15 11:53:10,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-15 11:53:10,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-15 11:53:10,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-15 11:53:10,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-15 11:53:10,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-15 11:53:10,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-15 11:53:10,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-15 11:53:10,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-15 11:53:10,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-15 11:53:10,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-15 11:53:10,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-15 11:53:10,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-15 11:53:10,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-15 11:53:10,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-15 11:53:10,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-15 11:53:10,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-15 11:53:10,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-15 11:53:10,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-15 11:53:10,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-15 11:53:10,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-15 11:53:10,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-15 11:53:10,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-15 11:53:10,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-15 11:53:10,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-15 11:53:10,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-15 11:53:10,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-15 11:53:10,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-15 11:53:10,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-15 11:53:10,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-15 11:53:10,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-15 11:53:10,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-15 11:53:10,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-15 11:53:10,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-15 11:53:10,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-15 11:53:10,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-15 11:53:10,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-15 11:53:10,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-15 11:53:10,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-15 11:53:10,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-15 11:53:10,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-15 11:53:10,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-15 11:53:10,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-15 11:53:10,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-15 11:53:10,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-15 11:53:10,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-15 11:53:10,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-15 11:53:10,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-15 11:53:10,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-15 11:53:10,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-15 11:53:10,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-15 11:53:10,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-15 11:53:10,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-15 11:53:10,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-15 11:53:10,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-15 11:53:10,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-15 11:53:10,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-15 11:53:10,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-15 11:53:10,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-15 11:53:10,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-15 11:53:10,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-15 11:53:10,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-15 11:53:10,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-15 11:53:10,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-15 11:53:10,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-15 11:53:10,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-15 11:53:10,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-15 11:53:10,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-15 11:53:10,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-15 11:53:10,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-15 11:53:10,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-15 11:53:10,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-15 11:53:10,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-15 11:53:10,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-15 11:53:10,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-15 11:53:10,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-15 11:53:10,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-15 11:53:10,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-15 11:53:10,706 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-15 11:53:10,707 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-15 11:53:10,707 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-15 11:53:11,155 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 760 ms to try 5000 different iterations. Found a solution that moves 4672 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18222263508600017. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03801437760048378, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06021100197387593, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5191111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03801437760048378, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1775, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24366666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-15 11:53:11,199 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv182938569:82 , srv1255774301:83 , srv1822198475:83 , srv2036591050:83 , srv328640100:83 , srv660559289:83 , srv902196171:83 , srv123010808:84 , srv2082418442:84 , srv523472923:84 , srv931320642:84 , srv126970828:85 , srv1338248177:85 , srv1457370374:85 , srv1514079114:85 , srv1527971759:85 , srv1583983527:85 , srv1736293470:85 , srv1753002889:85 , srv1770459162:85 , srv1864672934:85 , srv1908770240:85 , srv1946457003:85 , srv1992945499:85 , srv2073952959:85 , srv396784793:85 , srv400744726:85 , srv696545221:85 , srv825219335:85 , srv850901261:85 , srv894096847:85 , srv903976379:85 , srv921051127:85 , srv1080505292:86 , srv1096541632:86 , srv1259491330:86 , srv1277373608:86 , srv1292360648:86 , srv1314238935:86 , srv1325342040:86 , srv1474832259:86 , srv1526683479:86 , srv1708699763:86 , srv1754954443:86 , srv1763569383:86 , srv1773437254:86 , srv2087672772:86 , srv310979103:86 , srv547231258:86 , srv611426675:86 , srv724179225:86 , srv730521589:86 , srv737957241:86 , srv755062870:86 , srv814463233:86 , srv981942602:86 , srv1182566185:87 , srv1263971624:87 , srv1474549858:87 , srv1522329372:87 , srv1575028851:87 , srv2039224540:87 , srv2060225161:87 , srv2137138743:87 , srv24288931:87 , srv273168840:87 , srv290517432:87 , srv330195515:87 , srv368125431:87 , srv617160168:87 , srv721541665:87 , srv79986197:87 , srv96856434:87 , srv990548794:87 , srv113462942:88 , srv1803488849:88 , srv238710898:88 , srv482173621:88 , srv612249308:88 , srv656386454:88 , srv1210555252:89 , srv1792004007:89 , srv2097571299:89 , srv765865784:89 , srv79750131:89 , srv839511415:89 , srv848979275:89 , srv945804813:89 , srv1297686668:90 , srv1501393740:90 , srv2081676281:90 , srv1126309055:91 , srv1571047453:91 , srv1679041824:91 , srv431626909:91 , srv493457165:91 , srv873482528:91 , srv1168839644:93 , srv335475499:93 , srv1411080676:428 } 2023-08-15 11:53:11,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-15 11:53:11,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2097571299=53, srv79986197=84, srv990548794=99, srv1753002889=33, srv1314238935=15, srv1679041824=30, srv123010808=7, srv755062870=81, srv1763569383=35, srv2137138743=54, srv1263971624=10, srv1571047453=27, srv1754954443=34, srv400744726=65, srv1526683479=25, srv1708699763=31, srv113462942=3, srv24288931=56, srv2060225161=48, srv1474832259=21, srv1527971759=26, srv612249308=72, srv79750131=83, srv96856434=97, srv1864672934=42, srv2082418442=51, srv523472923=69, srv1770459162=36, srv1575028851=28, srv1080505292=0, srv696545221=76, srv1803488849=39, srv721541665=77, srv1946457003=44, srv765865784=82, srv368125431=63, srv1096541632=1, srv1338248177=17, srv126970828=11, srv1992945499=45, srv903976379=93, srv1210555252=6, srv1736293470=32, srv1292360648=13, srv902196171=92, srv931320642=95, srv396784793=64, srv1411080676=18, srv238710898=55, srv981942602=98, srv1474549858=20, srv730521589=79, srv1908770240=43, srv2039224540=47, srv1514079114=23, srv335475499=62, srv1297686668=14, srv1255774301=8, srv2073952959=49, srv1277373608=12, srv2081676281=50, srv848979275=88, srv1457370374=19, srv2036591050=46, srv656386454=74, srv431626909=66, srv724179225=78, srv1822198475=40, srv1522329372=24, srv330195515=61, srv814463233=85, srv1259491330=9, srv1182566185=5, srv1168839644=4, srv547231258=70, srv1126309055=2, srv921051127=94, srv1325342040=16, srv482173621=67, srv1501393740=22, srv617160168=73, srv894096847=91, srv182938569=41, srv273168840=57, srv873482528=90, srv660559289=75, srv1792004007=38, srv2087672772=52, srv825219335=86, srv290517432=58, srv945804813=96, srv850901261=89, srv1583983527=29, srv328640100=60, srv611426675=71, srv1773437254=37, srv737957241=80, srv493457165=68, srv839511415=87, srv310979103=59} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-15 11:53:11,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-15 11:53:11,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-15 11:53:11,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-15 11:53:11,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-15 11:53:11,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-15 11:53:11,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-15 11:53:11,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-15 11:53:11,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-15 11:53:11,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-15 11:53:11,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-15 11:53:11,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-15 11:53:11,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-15 11:53:11,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-15 11:53:11,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-15 11:53:11,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-15 11:53:11,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-15 11:53:11,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-15 11:53:11,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-15 11:53:11,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-15 11:53:11,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-15 11:53:11,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-15 11:53:11,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-15 11:53:11,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-15 11:53:11,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-15 11:53:11,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-15 11:53:11,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-15 11:53:11,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-15 11:53:11,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-15 11:53:11,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-15 11:53:11,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-15 11:53:11,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-15 11:53:11,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-15 11:53:11,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-15 11:53:11,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-15 11:53:11,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-15 11:53:11,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-15 11:53:11,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-15 11:53:11,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-15 11:53:11,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-15 11:53:11,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-15 11:53:11,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-15 11:53:11,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-15 11:53:11,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-15 11:53:11,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-15 11:53:11,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-15 11:53:11,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-15 11:53:11,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-15 11:53:11,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-15 11:53:11,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-15 11:53:11,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-15 11:53:11,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-15 11:53:11,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-15 11:53:11,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-15 11:53:11,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-15 11:53:11,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-15 11:53:11,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-15 11:53:11,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-15 11:53:11,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-15 11:53:11,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-15 11:53:11,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-15 11:53:11,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-15 11:53:11,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-15 11:53:11,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-15 11:53:11,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-15 11:53:11,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-15 11:53:11,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-15 11:53:11,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-15 11:53:11,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-15 11:53:11,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-15 11:53:11,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-15 11:53:11,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-15 11:53:11,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-15 11:53:11,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-15 11:53:11,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-15 11:53:11,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-15 11:53:11,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-15 11:53:11,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-15 11:53:11,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-15 11:53:11,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-15 11:53:11,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-15 11:53:11,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-15 11:53:11,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-15 11:53:11,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-15 11:53:11,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-15 11:53:11,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-15 11:53:11,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-15 11:53:11,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-15 11:53:11,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-15 11:53:11,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-15 11:53:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-15 11:53:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-15 11:53:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-15 11:53:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-15 11:53:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-15 11:53:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-15 11:53:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-15 11:53:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-15 11:53:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-15 11:53:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-15 11:53:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-15 11:53:11,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-15 11:53:11,428 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03801437760048378, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06021100197387593, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03801437760048378, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1775, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24366666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-15 11:53:11,428 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-15 11:53:11,429 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18218992104278892, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03801437760048378, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06021100197387593, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03801437760048378, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1775, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24366666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-15 11:53:11,795 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 517 ms to try 5000 different iterations. Found a solution that moves 2631 regions; Going from a computed imbalance of 0.18218992104278892 to a new imbalance of 0.015290896498172528. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023857874054142785); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.043979895417086516, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29233333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023857874054142785); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.047, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-15 11:53:11,805 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1679041824:78 , srv1571047453:79 , srv493457165:79 , srv1126309055:80 , srv113462942:80 , srv2097571299:80 , srv335475499:80 , srv1168839644:81 , srv1501393740:81 , srv2081676281:81 , srv839511415:81 , srv1263971624:82 , srv1297686668:82 , srv1575028851:82 , srv1792004007:82 , srv431626909:82 , srv79750131:82 , srv945804813:82 , srv1182566185:83 , srv1210555252:83 , srv238710898:83 , srv656386454:83 , srv765865784:83 , srv96856434:83 , srv1325342040:84 , srv1522329372:84 , srv1754954443:84 , srv1773437254:84 , srv1803488849:84 , srv273168840:84 , srv848979275:84 , srv873482528:84 , srv1096541632:85 , srv1259491330:85 , srv1292360648:85 , srv1474549858:85 , srv1514079114:85 , srv1708699763:85 , srv1736293470:85 , srv1753002889:85 , srv24288931:85 , srv330195515:85 , srv368125431:85 , srv612249308:85 , srv721541665:85 , srv850901261:85 , srv921051127:85 , srv1080505292:86 , srv126970828:86 , srv1277373608:86 , srv1457370374:86 , srv1526683479:86 , srv1583983527:86 , srv1908770240:86 , srv2039224540:86 , srv2060225161:86 , srv2073952959:86 , srv310979103:86 , srv547231258:86 , srv617160168:86 , srv724179225:86 , srv903976379:86 , srv931320642:86 , srv990548794:86 , srv1255774301:87 , srv1314238935:87 , srv1527971759:87 , srv1763569383:87 , srv1770459162:87 , srv1822198475:87 , srv2082418442:87 , srv2087672772:87 , srv290517432:87 , srv396784793:87 , srv482173621:87 , srv611426675:87 , srv696545221:87 , srv814463233:87 , srv981942602:87 , srv123010808:88 , srv1474832259:88 , srv1864672934:88 , srv1946457003:88 , srv1992945499:88 , srv2137138743:88 , srv737957241:88 , srv755062870:88 , srv79986197:88 , srv825219335:88 , srv1338248177:89 , srv182938569:89 , srv660559289:89 , srv730521589:89 , srv894096847:89 , srv2036591050:90 , srv328640100:90 , srv400744726:90 , srv523472923:90 , srv902196171:91 , srv1411080676:558 } 2023-08-15 11:53:11,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-15 11:53:11,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2097571299=53, srv79986197=84, srv990548794=99, srv1753002889=33, srv1314238935=15, srv1679041824=30, srv123010808=7, srv755062870=81, srv1763569383=35, srv2137138743=54, srv1263971624=10, srv1571047453=27, srv1754954443=34, srv400744726=65, srv1526683479=25, srv1708699763=31, srv113462942=3, srv24288931=56, srv2060225161=48, srv1474832259=21, srv1527971759=26, srv612249308=72, srv79750131=83, srv96856434=97, srv1864672934=42, srv2082418442=51, srv523472923=69, srv1770459162=36, srv1575028851=28, srv1080505292=0, srv696545221=76, srv1803488849=39, srv721541665=77, srv1946457003=44, srv765865784=82, srv368125431=63, srv1096541632=1, srv1338248177=17, srv126970828=11, srv1992945499=45, srv903976379=93, srv1210555252=6, srv1736293470=32, srv1292360648=13, srv902196171=92, srv931320642=95, srv396784793=64, srv1411080676=18, srv238710898=55, srv981942602=98, srv1474549858=20, srv730521589=79, srv1908770240=43, srv2039224540=47, srv1514079114=23, srv335475499=62, srv1297686668=14, srv1255774301=8, srv2073952959=49, srv1277373608=12, srv2081676281=50, srv848979275=88, srv1457370374=19, srv2036591050=46, srv656386454=74, srv431626909=66, srv724179225=78, srv1822198475=40, srv1522329372=24, srv330195515=61, srv814463233=85, srv1259491330=9, srv1182566185=5, srv1168839644=4, srv547231258=70, srv1126309055=2, srv921051127=94, srv1325342040=16, srv482173621=67, srv1501393740=22, srv617160168=73, srv894096847=91, srv182938569=41, srv273168840=57, srv873482528=90, srv660559289=75, srv1792004007=38, srv2087672772=52, srv825219335=86, srv290517432=58, srv945804813=96, srv850901261=89, srv1583983527=29, srv328640100=60, srv611426675=71, srv1773437254=37, srv737957241=80, srv493457165=68, srv839511415=87, srv310979103=59} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-15 11:53:11,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-15 11:53:11,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-15 11:53:11,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-15 11:53:11,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-15 11:53:11,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-15 11:53:11,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-15 11:53:11,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-15 11:53:11,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-15 11:53:11,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-15 11:53:11,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-15 11:53:11,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-15 11:53:11,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-15 11:53:11,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-15 11:53:11,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-15 11:53:11,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-15 11:53:11,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-15 11:53:11,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-15 11:53:11,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-15 11:53:11,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-15 11:53:11,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-15 11:53:11,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-15 11:53:11,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-15 11:53:11,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-15 11:53:11,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-15 11:53:11,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-15 11:53:11,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-15 11:53:11,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-15 11:53:11,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-15 11:53:11,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-15 11:53:11,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-15 11:53:11,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-15 11:53:11,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-15 11:53:11,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-15 11:53:11,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-15 11:53:11,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-15 11:53:11,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-15 11:53:11,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-15 11:53:11,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-15 11:53:11,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-15 11:53:11,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-15 11:53:11,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-15 11:53:11,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-15 11:53:11,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-15 11:53:11,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-15 11:53:11,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-15 11:53:11,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-15 11:53:11,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-15 11:53:11,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-15 11:53:11,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-15 11:53:11,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-15 11:53:11,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-15 11:53:11,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-15 11:53:11,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-15 11:53:11,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-15 11:53:11,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-15 11:53:11,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-15 11:53:11,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-15 11:53:11,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-15 11:53:11,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-15 11:53:11,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-15 11:53:11,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-15 11:53:11,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-15 11:53:11,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-15 11:53:11,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-15 11:53:11,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-15 11:53:11,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-15 11:53:11,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-15 11:53:11,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-15 11:53:11,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-15 11:53:11,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-15 11:53:11,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-15 11:53:11,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-15 11:53:11,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-15 11:53:11,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-15 11:53:11,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-15 11:53:11,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-15 11:53:11,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-15 11:53:11,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-15 11:53:11,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-15 11:53:11,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-15 11:53:11,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-15 11:53:11,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-15 11:53:11,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-15 11:53:11,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-15 11:53:11,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-15 11:53:11,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-15 11:53:11,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-15 11:53:11,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-15 11:53:11,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-15 11:53:11,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-15 11:53:11,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-15 11:53:11,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-15 11:53:11,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-15 11:53:11,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-15 11:53:11,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-15 11:53:11,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-15 11:53:11,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-15 11:53:11,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-15 11:53:11,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-15 11:53:11,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-15 11:53:11,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-15 11:53:12,014 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023857874054142785); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.043979895417086516, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023857874054142785); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.047, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-15 11:53:12,015 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-15 11:53:12,015 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015272473842417209, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023857874054142785); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.043979895417086516, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023857874054142785); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.047, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-15 11:53:12,377 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 508 ms to try 5000 different iterations. Found a solution that moves 1042 regions; Going from a computed imbalance of 0.015272473842417209 to a new imbalance of 6.855648301934767E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0064460309978613736); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.012116160563534423); MoveCostFunction : (multiplier=7.0, imbalance=0.11577777777777777, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0064460309978613736); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-15 11:53:12,381 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv79986197:81 , srv873482528:82 , srv1946457003:83 , srv2087672772:83 , srv2137138743:83 , srv290517432:83 , srv400744726:83 , srv730521589:83 , srv737957241:83 , srv755062870:83 , srv902196171:83 , srv1168839644:84 , srv1338248177:84 , srv1474832259:84 , srv1763569383:84 , srv1770459162:84 , srv1864672934:84 , srv2036591050:84 , srv2039224540:84 , srv2060225161:84 , srv328640100:84 , srv431626909:84 , srv482173621:84 , srv523472923:84 , srv612249308:84 , srv617160168:84 , srv894096847:84 , srv981942602:84 , srv123010808:85 , srv1314238935:85 , srv1474549858:85 , srv182938569:85 , srv1992945499:85 , srv2073952959:85 , srv2081676281:85 , srv24288931:85 , srv330195515:85 , srv335475499:85 , srv368125431:85 , srv396784793:85 , srv547231258:85 , srv611426675:85 , srv696545221:85 , srv721541665:85 , srv765865784:85 , srv814463233:85 , srv825219335:85 , srv848979275:85 , srv1080505292:86 , srv1126309055:86 , srv1210555252:86 , srv1259491330:86 , srv1277373608:86 , srv1292360648:86 , srv1297686668:86 , srv1501393740:86 , srv1527971759:86 , srv1583983527:86 , srv1792004007:86 , srv2082418442:86 , srv238710898:86 , srv273168840:86 , srv310979103:86 , srv656386454:86 , srv660559289:86 , srv724179225:86 , srv79750131:86 , srv903976379:86 , srv945804813:86 , srv990548794:86 , srv1182566185:87 , srv1255774301:87 , srv126970828:87 , srv1325342040:87 , srv1457370374:87 , srv1514079114:87 , srv1522329372:87 , srv1526683479:87 , srv1708699763:87 , srv1736293470:87 , srv1773437254:87 , srv1803488849:87 , srv1822198475:87 , srv850901261:87 , srv1096541632:88 , srv1263971624:88 , srv1571047453:88 , srv1575028851:88 , srv1753002889:88 , srv1754954443:88 , srv1908770240:88 , srv2097571299:88 , srv493457165:88 , srv839511415:88 , srv921051127:88 , srv931320642:88 , srv96856434:88 , srv1679041824:89 , srv113462942:90 , srv1411080676:529 } 2023-08-15 11:53:12,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-15 11:53:12,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2097571299=53, srv79986197=84, srv990548794=99, srv1753002889=33, srv1314238935=15, srv1679041824=30, srv123010808=7, srv755062870=81, srv1763569383=35, srv2137138743=54, srv1263971624=10, srv1571047453=27, srv1754954443=34, srv400744726=65, srv1526683479=25, srv1708699763=31, srv113462942=3, srv24288931=56, srv2060225161=48, srv1474832259=21, srv1527971759=26, srv612249308=72, srv79750131=83, srv96856434=97, srv1864672934=42, srv2082418442=51, srv523472923=69, srv1770459162=36, srv1575028851=28, srv1080505292=0, srv696545221=76, srv1803488849=39, srv721541665=77, srv1946457003=44, srv765865784=82, srv368125431=63, srv1096541632=1, srv1338248177=17, srv126970828=11, srv1992945499=45, srv903976379=93, srv1210555252=6, srv1736293470=32, srv1292360648=13, srv902196171=92, srv931320642=95, srv396784793=64, srv1411080676=18, srv238710898=55, srv981942602=98, srv1474549858=20, srv730521589=79, srv1908770240=43, srv2039224540=47, srv1514079114=23, srv335475499=62, srv1297686668=14, srv1255774301=8, srv2073952959=49, srv1277373608=12, srv2081676281=50, srv848979275=88, srv1457370374=19, srv2036591050=46, srv656386454=74, srv431626909=66, srv724179225=78, srv1822198475=40, srv1522329372=24, srv330195515=61, srv814463233=85, srv1259491330=9, srv1182566185=5, srv1168839644=4, srv547231258=70, srv1126309055=2, srv921051127=94, srv1325342040=16, srv482173621=67, srv1501393740=22, srv617160168=73, srv894096847=91, srv182938569=41, srv273168840=57, srv873482528=90, srv660559289=75, srv1792004007=38, srv2087672772=52, srv825219335=86, srv290517432=58, srv945804813=96, srv850901261=89, srv1583983527=29, srv328640100=60, srv611426675=71, srv1773437254=37, srv737957241=80, srv493457165=68, srv839511415=87, srv310979103=59} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-15 11:53:12,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-15 11:53:12,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-15 11:53:12,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-15 11:53:12,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-15 11:53:12,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-15 11:53:12,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-15 11:53:12,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-15 11:53:12,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-15 11:53:12,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-15 11:53:12,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-15 11:53:12,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-15 11:53:12,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-15 11:53:12,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-15 11:53:12,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-15 11:53:12,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-15 11:53:12,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-15 11:53:12,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-15 11:53:12,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-15 11:53:12,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-15 11:53:12,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-15 11:53:12,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-15 11:53:12,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-15 11:53:12,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-15 11:53:12,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-15 11:53:12,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-15 11:53:12,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-15 11:53:12,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-15 11:53:12,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-15 11:53:12,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-15 11:53:12,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-15 11:53:12,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-15 11:53:12,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-15 11:53:12,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-15 11:53:12,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-15 11:53:12,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-15 11:53:12,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-15 11:53:12,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-15 11:53:12,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-15 11:53:12,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-15 11:53:12,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-15 11:53:12,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-15 11:53:12,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-15 11:53:12,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-15 11:53:12,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-15 11:53:12,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-15 11:53:12,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-15 11:53:12,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-15 11:53:12,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-15 11:53:12,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-15 11:53:12,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-15 11:53:12,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-15 11:53:12,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-15 11:53:12,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-15 11:53:12,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-15 11:53:12,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-15 11:53:12,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-15 11:53:12,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-15 11:53:12,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-15 11:53:12,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-15 11:53:12,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-15 11:53:12,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-15 11:53:12,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-15 11:53:12,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-15 11:53:12,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-15 11:53:12,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-15 11:53:12,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-15 11:53:12,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-15 11:53:12,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-15 11:53:12,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-15 11:53:12,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-15 11:53:12,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-15 11:53:12,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-15 11:53:12,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-15 11:53:12,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-15 11:53:12,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-15 11:53:12,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-15 11:53:12,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-15 11:53:12,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-15 11:53:12,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-15 11:53:12,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-15 11:53:12,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-15 11:53:12,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-15 11:53:12,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-15 11:53:12,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-15 11:53:12,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-15 11:53:12,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-15 11:53:12,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-15 11:53:12,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-15 11:53:12,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-15 11:53:12,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-15 11:53:12,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-15 11:53:12,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-15 11:53:12,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-15 11:53:12,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-15 11:53:12,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-15 11:53:12,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-15 11:53:12,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-15 11:53:12,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-15 11:53:12,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-15 11:53:12,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-15 11:53:12,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-15 11:53:12,625 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0064460309978613736); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.012116160563534423); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0064460309978613736); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-15 11:53:12,625 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-15 11:53:12,626 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.782685902478134E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0064460309978613736); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.012116160563534423); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0064460309978613736); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-15 11:53:12,973 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 492 ms to try 5000 different iterations. Found a solution that moves 248 regions; Going from a computed imbalance of 6.782685902478134E-4 to a new imbalance of 3.774066766423061E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002551379987942442); MoveCostFunction : (multiplier=7.0, imbalance=0.027555555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-15 11:53:12,974 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1096541632:83 , srv1526683479:83 , srv1754954443:83 , srv1803488849:83 , srv482173621:83 , srv921051127:83 , srv1080505292:84 , srv1126309055:84 , srv113462942:84 , srv1168839644:84 , srv1210555252:84 , srv126970828:84 , srv1277373608:84 , srv1297686668:84 , srv1314238935:84 , srv1338248177:84 , srv1457370374:84 , srv1474832259:84 , srv1501393740:84 , srv1522329372:84 , srv1527971759:84 , srv1571047453:84 , srv1679041824:84 , srv1708699763:84 , srv1753002889:84 , srv1792004007:84 , srv1864672934:84 , srv1908770240:84 , srv1992945499:84 , srv2036591050:84 , srv2137138743:84 , srv310979103:84 , srv335475499:84 , srv493457165:84 , srv523472923:84 , srv611426675:84 , srv660559289:84 , srv724179225:84 , srv730521589:84 , srv814463233:84 , srv825219335:84 , srv839511415:84 , srv848979275:84 , srv894096847:84 , srv931320642:84 , srv96856434:84 , srv990548794:84 , srv1182566185:85 , srv123010808:85 , srv1255774301:85 , srv1259491330:85 , srv1263971624:85 , srv1292360648:85 , srv1325342040:85 , srv1474549858:85 , srv1514079114:85 , srv1575028851:85 , srv1583983527:85 , srv1736293470:85 , srv1763569383:85 , srv1773437254:85 , srv2039224540:85 , srv2060225161:85 , srv2082418442:85 , srv2097571299:85 , srv238710898:85 , srv24288931:85 , srv273168840:85 , srv290517432:85 , srv330195515:85 , srv368125431:85 , srv396784793:85 , srv400744726:85 , srv431626909:85 , srv547231258:85 , srv612249308:85 , srv617160168:85 , srv656386454:85 , srv696545221:85 , srv737957241:85 , srv765865784:85 , srv79750131:85 , srv850901261:85 , srv873482528:85 , srv902196171:85 , srv903976379:85 , srv945804813:85 , srv981942602:85 , srv1770459162:86 , srv1822198475:86 , srv182938569:86 , srv1946457003:86 , srv2073952959:86 , srv2081676281:86 , srv2087672772:86 , srv328640100:86 , srv721541665:86 , srv755062870:86 , srv79986197:86 , srv1411080676:627 } 2023-08-15 11:53:12,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-15 11:53:12,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2097571299=53, srv79986197=84, srv990548794=99, srv1753002889=33, srv1314238935=15, srv1679041824=30, srv123010808=7, srv755062870=81, srv1763569383=35, srv2137138743=54, srv1263971624=10, srv1571047453=27, srv1754954443=34, srv400744726=65, srv1526683479=25, srv1708699763=31, srv113462942=3, srv24288931=56, srv2060225161=48, srv1474832259=21, srv1527971759=26, srv612249308=72, srv79750131=83, srv96856434=97, srv1864672934=42, srv2082418442=51, srv523472923=69, srv1770459162=36, srv1575028851=28, srv1080505292=0, srv696545221=76, srv1803488849=39, srv721541665=77, srv1946457003=44, srv765865784=82, srv368125431=63, srv1096541632=1, srv1338248177=17, srv126970828=11, srv1992945499=45, srv903976379=93, srv1210555252=6, srv1736293470=32, srv1292360648=13, srv902196171=92, srv931320642=95, srv396784793=64, srv1411080676=18, srv238710898=55, srv981942602=98, srv1474549858=20, srv730521589=79, srv1908770240=43, srv2039224540=47, srv1514079114=23, srv335475499=62, srv1297686668=14, srv1255774301=8, srv2073952959=49, srv1277373608=12, srv2081676281=50, srv848979275=88, srv1457370374=19, srv2036591050=46, srv656386454=74, srv431626909=66, srv724179225=78, srv1822198475=40, srv1522329372=24, srv330195515=61, srv814463233=85, srv1259491330=9, srv1182566185=5, srv1168839644=4, srv547231258=70, srv1126309055=2, srv921051127=94, srv1325342040=16, srv482173621=67, srv1501393740=22, srv617160168=73, srv894096847=91, srv182938569=41, srv273168840=57, srv873482528=90, srv660559289=75, srv1792004007=38, srv2087672772=52, srv825219335=86, srv290517432=58, srv945804813=96, srv850901261=89, srv1583983527=29, srv328640100=60, srv611426675=71, srv1773437254=37, srv737957241=80, srv493457165=68, srv839511415=87, srv310979103=59} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-15 11:53:12,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-15 11:53:12,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-15 11:53:12,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-15 11:53:12,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-15 11:53:12,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-15 11:53:12,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-15 11:53:12,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-15 11:53:12,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-15 11:53:12,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-15 11:53:12,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-15 11:53:12,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-15 11:53:12,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-15 11:53:12,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-15 11:53:12,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-15 11:53:12,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-15 11:53:12,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-15 11:53:12,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-15 11:53:12,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-15 11:53:12,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-15 11:53:12,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-15 11:53:12,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-15 11:53:12,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-15 11:53:12,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-15 11:53:12,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-15 11:53:12,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-15 11:53:12,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-15 11:53:12,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-15 11:53:12,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-15 11:53:12,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-15 11:53:12,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-15 11:53:12,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-15 11:53:12,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-15 11:53:12,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-15 11:53:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-15 11:53:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-15 11:53:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-15 11:53:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-15 11:53:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-15 11:53:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-15 11:53:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-15 11:53:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-15 11:53:12,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-15 11:53:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-15 11:53:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-15 11:53:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-15 11:53:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-15 11:53:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-15 11:53:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-15 11:53:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-15 11:53:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-15 11:53:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-15 11:53:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-15 11:53:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-15 11:53:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-15 11:53:12,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-15 11:53:12,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-15 11:53:12,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-15 11:53:12,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-15 11:53:12,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-15 11:53:12,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-15 11:53:12,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-15 11:53:12,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-15 11:53:12,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-15 11:53:12,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-15 11:53:12,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-15 11:53:12,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-15 11:53:12,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-15 11:53:12,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-15 11:53:12,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-15 11:53:12,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-15 11:53:12,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-15 11:53:12,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-15 11:53:12,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-15 11:53:12,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-15 11:53:12,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-15 11:53:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-15 11:53:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-15 11:53:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-15 11:53:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-15 11:53:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-15 11:53:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-15 11:53:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-15 11:53:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-15 11:53:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-15 11:53:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-15 11:53:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-15 11:53:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-15 11:53:12,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-15 11:53:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-15 11:53:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-15 11:53:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-15 11:53:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-15 11:53:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-15 11:53:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-15 11:53:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-15 11:53:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-15 11:53:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-15 11:53:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-15 11:53:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-15 11:53:12,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-15 11:53:12,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-15 11:53:13,136 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002551379987942442); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-15 11:53:13,136 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-15 11:53:13,137 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.60041345485641E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002551379987942442); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-15 11:53:13,456 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 432 ms to try 5000 different iterations. Found a solution that moves 39 regions; Going from a computed imbalance of 3.60041345485641E-5 to a new imbalance of 8.252773953778466E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.004333333333333333); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-15 11:53:13,457 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1822198475:83 , srv1908770240:83 , srv328640100:83 , srv400744726:83 , srv721541665:83 , srv755062870:83 , srv990548794:83 , srv1080505292:84 , srv1096541632:84 , srv113462942:84 , srv1168839644:84 , srv1182566185:84 , srv1210555252:84 , srv123010808:84 , srv1255774301:84 , srv1259491330:84 , srv1263971624:84 , srv126970828:84 , srv1277373608:84 , srv1292360648:84 , srv1297686668:84 , srv1314238935:84 , srv1325342040:84 , srv1338248177:84 , srv1457370374:84 , srv1474549858:84 , srv1474832259:84 , srv1514079114:84 , srv1522329372:84 , srv1526683479:84 , srv1527971759:84 , srv1571047453:84 , srv1575028851:84 , srv1583983527:84 , srv1679041824:84 , srv1708699763:84 , srv1736293470:84 , srv1753002889:84 , srv1763569383:84 , srv1770459162:84 , srv1773437254:84 , srv1803488849:84 , srv182938569:84 , srv1946457003:84 , srv1992945499:84 , srv2039224540:84 , srv2060225161:84 , srv2073952959:84 , srv2081676281:84 , srv2082418442:84 , srv2087672772:84 , srv2097571299:84 , srv2137138743:84 , srv238710898:84 , srv24288931:84 , srv273168840:84 , srv290517432:84 , srv310979103:84 , srv330195515:84 , srv335475499:84 , srv368125431:84 , srv396784793:84 , srv431626909:84 , srv482173621:84 , srv493457165:84 , srv523472923:84 , srv547231258:84 , srv611426675:84 , srv612249308:84 , srv617160168:84 , srv656386454:84 , srv660559289:84 , srv696545221:84 , srv724179225:84 , srv730521589:84 , srv737957241:84 , srv765865784:84 , srv79750131:84 , srv79986197:84 , srv814463233:84 , srv825219335:84 , srv839511415:84 , srv848979275:84 , srv850901261:84 , srv873482528:84 , srv894096847:84 , srv902196171:84 , srv903976379:84 , srv931320642:84 , srv945804813:84 , srv96856434:84 , srv981942602:84 , srv1126309055:85 , srv1501393740:85 , srv1754954443:85 , srv1792004007:85 , srv1864672934:85 , srv2036591050:85 , srv921051127:85 , srv1411080676:684 } 2023-08-15 11:53:13,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-15 11:53:13,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2097571299=53, srv79986197=84, srv990548794=99, srv1753002889=33, srv1314238935=15, srv1679041824=30, srv123010808=7, srv755062870=81, srv1763569383=35, srv2137138743=54, srv1263971624=10, srv1571047453=27, srv1754954443=34, srv400744726=65, srv1526683479=25, srv1708699763=31, srv113462942=3, srv24288931=56, srv2060225161=48, srv1474832259=21, srv1527971759=26, srv612249308=72, srv79750131=83, srv96856434=97, srv1864672934=42, srv2082418442=51, srv523472923=69, srv1770459162=36, srv1575028851=28, srv1080505292=0, srv696545221=76, srv1803488849=39, srv721541665=77, srv1946457003=44, srv765865784=82, srv368125431=63, srv1096541632=1, srv1338248177=17, srv126970828=11, srv1992945499=45, srv903976379=93, srv1210555252=6, srv1736293470=32, srv1292360648=13, srv902196171=92, srv931320642=95, srv396784793=64, srv1411080676=18, srv238710898=55, srv981942602=98, srv1474549858=20, srv730521589=79, srv1908770240=43, srv2039224540=47, srv1514079114=23, srv335475499=62, srv1297686668=14, srv1255774301=8, srv2073952959=49, srv1277373608=12, srv2081676281=50, srv848979275=88, srv1457370374=19, srv2036591050=46, srv656386454=74, srv431626909=66, srv724179225=78, srv1822198475=40, srv1522329372=24, srv330195515=61, srv814463233=85, srv1259491330=9, srv1182566185=5, srv1168839644=4, srv547231258=70, srv1126309055=2, srv921051127=94, srv1325342040=16, srv482173621=67, srv1501393740=22, srv617160168=73, srv894096847=91, srv182938569=41, srv273168840=57, srv873482528=90, srv660559289=75, srv1792004007=38, srv2087672772=52, srv825219335=86, srv290517432=58, srv945804813=96, srv850901261=89, srv1583983527=29, srv328640100=60, srv611426675=71, srv1773437254=37, srv737957241=80, srv493457165=68, srv839511415=87, srv310979103=59} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-15 11:53:13,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-15 11:53:13,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-15 11:53:13,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-15 11:53:13,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-15 11:53:13,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-15 11:53:13,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-15 11:53:13,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-15 11:53:13,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-15 11:53:13,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-15 11:53:13,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-15 11:53:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-15 11:53:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-15 11:53:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-15 11:53:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-15 11:53:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-15 11:53:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-15 11:53:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-15 11:53:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-15 11:53:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-15 11:53:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-15 11:53:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-15 11:53:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-15 11:53:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-15 11:53:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-15 11:53:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-15 11:53:13,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-15 11:53:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-15 11:53:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-15 11:53:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-15 11:53:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-15 11:53:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-15 11:53:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-15 11:53:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-15 11:53:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-15 11:53:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-15 11:53:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-15 11:53:13,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-15 11:53:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-15 11:53:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-15 11:53:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-15 11:53:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-15 11:53:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-15 11:53:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-15 11:53:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-15 11:53:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-15 11:53:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-15 11:53:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-15 11:53:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-15 11:53:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-15 11:53:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-15 11:53:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-15 11:53:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-15 11:53:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-15 11:53:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-15 11:53:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-15 11:53:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-15 11:53:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-15 11:53:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-15 11:53:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-15 11:53:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-15 11:53:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-15 11:53:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-15 11:53:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-15 11:53:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-15 11:53:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-15 11:53:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-15 11:53:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-15 11:53:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-15 11:53:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-15 11:53:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-15 11:53:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-15 11:53:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-15 11:53:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-15 11:53:13,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-15 11:53:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-15 11:53:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-15 11:53:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-15 11:53:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-15 11:53:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-15 11:53:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-15 11:53:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-15 11:53:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-15 11:53:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-15 11:53:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-15 11:53:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-15 11:53:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-15 11:53:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-15 11:53:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-15 11:53:13,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-15 11:53:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-15 11:53:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-15 11:53:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-15 11:53:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-15 11:53:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-15 11:53:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-15 11:53:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-15 11:53:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-15 11:53:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-15 11:53:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-15 11:53:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-15 11:53:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-15 11:53:13,614 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.979690117040589E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-15 11:53:13,614 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1822198475:83 , srv1908770240:83 , srv328640100:83 , srv400744726:83 , srv721541665:83 , srv755062870:83 , srv990548794:83 , srv1080505292:84 , srv1096541632:84 , srv113462942:84 , srv1168839644:84 , srv1182566185:84 , srv1210555252:84 , srv123010808:84 , srv1255774301:84 , srv1259491330:84 , srv1263971624:84 , srv126970828:84 , srv1277373608:84 , srv1292360648:84 , srv1297686668:84 , srv1314238935:84 , srv1325342040:84 , srv1338248177:84 , srv1457370374:84 , srv1474549858:84 , srv1474832259:84 , srv1514079114:84 , srv1522329372:84 , srv1526683479:84 , srv1527971759:84 , srv1571047453:84 , srv1575028851:84 , srv1583983527:84 , srv1679041824:84 , srv1708699763:84 , srv1736293470:84 , srv1753002889:84 , srv1763569383:84 , srv1770459162:84 , srv1773437254:84 , srv1803488849:84 , srv182938569:84 , srv1946457003:84 , srv1992945499:84 , srv2039224540:84 , srv2060225161:84 , srv2073952959:84 , srv2081676281:84 , srv2082418442:84 , srv2087672772:84 , srv2097571299:84 , srv2137138743:84 , srv238710898:84 , srv24288931:84 , srv273168840:84 , srv290517432:84 , srv310979103:84 , srv330195515:84 , srv335475499:84 , srv368125431:84 , srv396784793:84 , srv431626909:84 , srv482173621:84 , srv493457165:84 , srv523472923:84 , srv547231258:84 , srv611426675:84 , srv612249308:84 , srv617160168:84 , srv656386454:84 , srv660559289:84 , srv696545221:84 , srv724179225:84 , srv730521589:84 , srv737957241:84 , srv765865784:84 , srv79750131:84 , srv79986197:84 , srv814463233:84 , srv825219335:84 , srv839511415:84 , srv848979275:84 , srv850901261:84 , srv873482528:84 , srv894096847:84 , srv902196171:84 , srv903976379:84 , srv931320642:84 , srv945804813:84 , srv96856434:84 , srv981942602:84 , srv1126309055:85 , srv1501393740:85 , srv1754954443:85 , srv1792004007:85 , srv1864672934:85 , srv2036591050:85 , srv921051127:85 , srv1411080676:684 } 2023-08-15 11:53:13,644 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=307 (was 264) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=2278 (was 3127) 2023-08-15 11:53:13,652 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=307, ProcessCount=171, AvailableMemoryMB=2278 2023-08-15 11:53:13,653 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-15 11:53:13,653 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-08-15 11:53:13,666 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1047146846:84 , srv1059087720:84 , srv1065188662:84 , srv1127825421:84 , srv1155239141:84 , srv1245170745:84 , srv1338060123:84 , srv1396510284:84 , srv1445023625:84 , srv1525454926:84 , srv1569564261:84 , srv1602236811:84 , srv1845646313:84 , srv1915052919:84 , srv1941405574:84 , srv346847519:84 , srv346945545:84 , srv363905260:84 , srv407362262:84 , srv510884111:84 , srv570835341:84 , srv71955714:84 , srv719954905:84 , srv81812177:84 , srv857213582:84 , srv865536764:84 , srv904789838:84 , srv924842503:84 , srv987689664:84 , srv1962882386:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-08-15 11:53:13,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-15 11:53:13,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1155239141=4, srv865536764=26, srv1338060123=6, srv71955714=22, srv904789838=27, srv1065188662=2, srv1602236811=11, srv1047146846=0, srv1127825421=3, srv1396510284=7, srv1845646313=12, srv346847519=16, srv1525454926=9, srv1569564261=10, srv924842503=28, srv1059087720=1, srv1941405574=14, srv81812177=24, srv570835341=21, srv1915052919=13, srv407362262=19, srv510884111=20, srv346945545=17, srv1245170745=5, srv857213582=25, srv987689664=29, srv363905260=18, srv719954905=23, srv1962882386=15, srv1445023625=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-15 11:53:13,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-15 11:53:13,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-15 11:53:13,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-15 11:53:13,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-15 11:53:13,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-15 11:53:13,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-15 11:53:13,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-15 11:53:13,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-15 11:53:13,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-15 11:53:13,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-15 11:53:13,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-15 11:53:13,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-15 11:53:13,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-15 11:53:13,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-15 11:53:13,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-15 11:53:13,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-15 11:53:13,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-15 11:53:13,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-15 11:53:13,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-15 11:53:13,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-15 11:53:13,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-15 11:53:13,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-15 11:53:13,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-15 11:53:13,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-15 11:53:13,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-15 11:53:13,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-15 11:53:13,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-15 11:53:13,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-15 11:53:13,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-15 11:53:13,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-15 11:53:13,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-15 11:53:13,700 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.084449054772645, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-15 11:53:13,701 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912459139478355, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.084449054772645, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-08-15 11:55:13,687 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5582945 different iterations. Found a solution that moves 1870 regions; Going from a computed imbalance of 0.9912459139478355 to a new imbalance of 4.3646732880327594E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6925925925925925, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-15 11:55:13,693 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1047146846:90 , srv1059087720:90 , srv1065188662:90 , srv1127825421:90 , srv1155239141:90 , srv1245170745:90 , srv1338060123:90 , srv1396510284:90 , srv1445023625:90 , srv1525454926:90 , srv1569564261:90 , srv1602236811:90 , srv1845646313:90 , srv1915052919:90 , srv1941405574:90 , srv1962882386:90 , srv346847519:90 , srv346945545:90 , srv363905260:90 , srv407362262:90 , srv510884111:90 , srv570835341:90 , srv71955714:90 , srv719954905:90 , srv81812177:90 , srv857213582:90 , srv865536764:90 , srv904789838:90 , srv924842503:90 , srv987689664:90 } 2023-08-15 11:55:13,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-15 11:55:13,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1155239141=4, srv865536764=26, srv1338060123=6, srv71955714=22, srv904789838=27, srv1065188662=2, srv1602236811=11, srv1047146846=0, srv1127825421=3, srv1396510284=7, srv1845646313=12, srv346847519=16, srv1525454926=9, srv1569564261=10, srv924842503=28, srv1059087720=1, srv1941405574=14, srv81812177=24, srv570835341=21, srv1915052919=13, srv407362262=19, srv510884111=20, srv346945545=17, srv1245170745=5, srv857213582=25, srv987689664=29, srv363905260=18, srv719954905=23, srv1962882386=15, srv1445023625=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-15 11:55:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-15 11:55:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-15 11:55:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-15 11:55:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-15 11:55:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-15 11:55:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-15 11:55:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-15 11:55:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-15 11:55:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-15 11:55:13,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-15 11:55:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-15 11:55:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-15 11:55:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-15 11:55:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-15 11:55:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-15 11:55:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-15 11:55:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-15 11:55:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-15 11:55:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-15 11:55:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-15 11:55:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-15 11:55:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-15 11:55:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-15 11:55:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-15 11:55:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-15 11:55:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-15 11:55:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-15 11:55:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-15 11:55:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-15 11:55:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-15 11:55:13,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-15 11:55:13,711 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-15 11:55:13,712 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1047146846:90 , srv1059087720:90 , srv1065188662:90 , srv1127825421:90 , srv1155239141:90 , srv1245170745:90 , srv1338060123:90 , srv1396510284:90 , srv1445023625:90 , srv1525454926:90 , srv1569564261:90 , srv1602236811:90 , srv1845646313:90 , srv1915052919:90 , srv1941405574:90 , srv1962882386:90 , srv346847519:90 , srv346945545:90 , srv363905260:90 , srv407362262:90 , srv510884111:90 , srv570835341:90 , srv71955714:90 , srv719954905:90 , srv81812177:90 , srv857213582:90 , srv865536764:90 , srv904789838:90 , srv924842503:90 , srv987689664:90 } 2023-08-15 11:55:13,724 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=200 (was 307), ProcessCount=172 (was 171) - ProcessCount LEAK? -, AvailableMemoryMB=2172 (was 2278)