2023-05-12 19:53:21,399 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-05-12 19:53:21,752 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-05-12 19:53:21,877 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-05-12 19:53:21,945 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-12 19:53:21,952 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-12 19:53:21,965 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=312, ProcessCount=175, AvailableMemoryMB=10391 2023-05-12 19:53:21,967 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-12 19:53:21,967 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-12 19:53:22,098 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1016565859:84 , srv1018733457:84 , srv1067036299:84 , srv1082357286:84 , srv1168093835:84 , srv1176282496:84 , srv1181549387:84 , srv1181749762:84 , srv118609237:84 , srv1193576727:84 , srv1194870565:84 , srv1227909756:84 , srv1276959087:84 , srv1289306039:84 , srv1316050746:84 , srv138639376:84 , srv1387433622:84 , srv139904411:84 , srv1400803415:84 , srv1403206514:84 , srv1428903602:84 , srv1455968853:84 , srv1459474246:84 , srv1488608033:84 , srv1506525301:84 , srv1547918752:84 , srv1613769265:84 , srv1628128231:84 , srv1637439305:84 , srv1692621681:84 , srv1693545208:84 , srv1714702987:84 , srv1720307728:84 , srv1724424879:84 , srv1735017133:84 , srv1819596117:84 , srv1829450491:84 , srv186663345:84 , srv1897235971:84 , srv1935796059:84 , srv1946049863:84 , srv1950779215:84 , srv1957365466:84 , srv1958400536:84 , srv1996367679:84 , srv2005349747:84 , srv204888190:84 , srv207376053:84 , srv2076031703:84 , srv2078967015:84 , srv2109871535:84 , srv2139226147:84 , srv227882650:84 , srv244896692:84 , srv258434666:84 , srv259964897:84 , srv274523191:84 , srv307228853:84 , srv309715874:84 , srv3105223:84 , srv33600481:84 , srv344921439:84 , srv348904254:84 , srv358102847:84 , srv372587919:84 , srv391763850:84 , srv42614937:84 , srv441872125:84 , srv454925965:84 , srv455078980:84 , srv463649486:84 , srv471362409:84 , srv473943284:84 , srv489411269:84 , srv527055038:84 , srv530735550:84 , srv531216229:84 , srv543492888:84 , srv57703892:84 , srv619612979:84 , srv642466468:84 , srv654579367:84 , srv65837124:84 , srv691507786:84 , srv700116442:84 , srv70105780:84 , srv716695538:84 , srv768347900:84 , srv791745173:84 , srv792242379:84 , srv823931606:84 , srv883653820:84 , srv913636735:84 , srv935887410:84 , srv936548419:84 , srv938651297:84 , srv954907228:84 , srv963597245:84 , srv993829056:84 , srv1093683352:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-05-12 19:53:22,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-12 19:53:22,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1403206514=20, srv274523191=57, srv700116442=85, srv2139226147=52, srv1897235971=39, srv792242379=90, srv118609237=9, srv531216229=77, srv1181549387=7, srv1289306039=14, srv2076031703=49, srv65837124=83, srv1016565859=0, srv1720307728=33, srv358102847=64, srv391763850=66, srv543492888=78, srv57703892=79, srv936548419=95, srv938651297=96, srv1168093835=5, srv3105223=60, srv913636735=93, srv1735017133=35, srv1194870565=11, srv1628128231=28, srv70105780=86, srv1613769265=27, srv1093683352=4, srv1819596117=36, srv2109871535=51, srv791745173=89, srv307228853=58, srv1693545208=31, srv344921439=62, srv1193576727=10, srv258434666=55, srv455078980=70, srv1935796059=40, srv2078967015=50, srv954907228=97, srv1946049863=41, srv1276959087=13, srv691507786=84, srv527055038=75, srv1547918752=26, srv227882650=53, srv204888190=47, srv42614937=67, srv1958400536=44, srv1724424879=34, srv372587919=65, srv1637439305=29, srv1714702987=32, srv993829056=99, srv139904411=18, srv489411269=74, srv1400803415=19, srv138639376=16, srv1459474246=23, srv1387433622=17, srv186663345=38, srv33600481=61, srv654579367=82, srv963597245=98, srv454925965=69, srv1018733457=1, srv530735550=76, srv768347900=88, srv1316050746=15, srv207376053=48, srv1506525301=25, srv1488608033=24, srv1692621681=30, srv1067036299=2, srv309715874=59, srv935887410=94, srv1082357286=3, srv883653820=92, srv1455968853=22, srv471362409=72, srv2005349747=46, srv619612979=80, srv463649486=71, srv1957365466=43, srv1996367679=45, srv244896692=54, srv823931606=91, srv1176282496=6, srv1950779215=42, srv1829450491=37, srv1181749762=8, srv348904254=63, srv259964897=56, srv642466468=81, srv1227909756=12, srv1428903602=21, srv473943284=73, srv716695538=87, srv441872125=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-12 19:53:22,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-12 19:53:22,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-12 19:53:22,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-12 19:53:22,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-12 19:53:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-12 19:53:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-12 19:53:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-12 19:53:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-12 19:53:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-12 19:53:22,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-12 19:53:22,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-12 19:53:22,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-12 19:53:22,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-12 19:53:22,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-12 19:53:22,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-12 19:53:22,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-12 19:53:22,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-12 19:53:22,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-12 19:53:22,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-12 19:53:22,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-12 19:53:22,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-12 19:53:22,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-12 19:53:22,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-12 19:53:22,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-12 19:53:22,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-12 19:53:22,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-12 19:53:22,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-12 19:53:22,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-12 19:53:22,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-12 19:53:22,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-12 19:53:22,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-12 19:53:22,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-12 19:53:22,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-12 19:53:22,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-12 19:53:22,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-12 19:53:22,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-12 19:53:22,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-12 19:53:22,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-12 19:53:22,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-12 19:53:22,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-12 19:53:22,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-12 19:53:22,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-12 19:53:22,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-12 19:53:22,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-12 19:53:22,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-12 19:53:22,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-12 19:53:22,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-12 19:53:22,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-12 19:53:22,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-12 19:53:22,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-12 19:53:22,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-12 19:53:22,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-12 19:53:22,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-12 19:53:22,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-12 19:53:22,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-12 19:53:22,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-12 19:53:22,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-12 19:53:22,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-12 19:53:22,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-12 19:53:22,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-12 19:53:22,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-12 19:53:22,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-12 19:53:22,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-12 19:53:22,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-12 19:53:22,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-12 19:53:22,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-12 19:53:22,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-12 19:53:22,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-12 19:53:22,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-12 19:53:22,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-12 19:53:22,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-12 19:53:22,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-12 19:53:22,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-12 19:53:22,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-12 19:53:22,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-12 19:53:22,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-12 19:53:22,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-12 19:53:22,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-12 19:53:22,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-12 19:53:22,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-12 19:53:22,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-12 19:53:22,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-12 19:53:22,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-12 19:53:22,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-12 19:53:22,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-12 19:53:22,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-12 19:53:22,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-12 19:53:22,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-12 19:53:22,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-12 19:53:22,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-12 19:53:22,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-12 19:53:22,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-12 19:53:22,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-12 19:53:22,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-12 19:53:22,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-12 19:53:22,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-12 19:53:22,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-12 19:53:22,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-12 19:53:22,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-12 19:53:22,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-12 19:53:22,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-12 19:53:22,539 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-12 19:53:22,539 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-12 19:53:22,539 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-12 19:53:23,293 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1027 ms to try 5000 different iterations. Found a solution that moves 4701 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18304891573455764. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03815190723854308, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.058580285349679136, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5223333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03815190723854308, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17858333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24208333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-12 19:53:23,335 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1946049863:82 , srv441872125:82 , srv454925965:82 , srv1016565859:83 , srv1067036299:83 , srv1181749762:83 , srv1387433622:83 , srv1735017133:83 , srv527055038:83 , srv1194870565:84 , srv1459474246:84 , srv1506525301:84 , srv1547918752:84 , srv227882650:84 , srv244896692:84 , srv344921439:84 , srv935887410:84 , srv1176282496:85 , srv1193576727:85 , srv1316050746:85 , srv138639376:85 , srv1692621681:85 , srv1693545208:85 , srv1958400536:85 , srv2076031703:85 , srv2078967015:85 , srv2139226147:85 , srv258434666:85 , srv33600481:85 , srv391763850:85 , srv42614937:85 , srv455078980:85 , srv531216229:85 , srv654579367:85 , srv691507786:85 , srv716695538:85 , srv768347900:85 , srv1018733457:86 , srv1227909756:86 , srv1276959087:86 , srv139904411:86 , srv1403206514:86 , srv1628128231:86 , srv1720307728:86 , srv204888190:86 , srv2109871535:86 , srv489411269:86 , srv642466468:86 , srv791745173:86 , srv936548419:86 , srv954907228:86 , srv1082357286:87 , srv1400803415:87 , srv1613769265:87 , srv1714702987:87 , srv1819596117:87 , srv1829450491:87 , srv186663345:87 , srv1957365466:87 , srv207376053:87 , srv3105223:87 , srv372587919:87 , srv471362409:87 , srv530735550:87 , srv70105780:87 , srv792242379:87 , srv913636735:87 , srv963597245:87 , srv1168093835:88 , srv1181549387:88 , srv1428903602:88 , srv1455968853:88 , srv1935796059:88 , srv1950779215:88 , srv2005349747:88 , srv259964897:88 , srv309715874:88 , srv348904254:88 , srv463649486:88 , srv543492888:88 , srv57703892:88 , srv619612979:88 , srv700116442:88 , srv1724424879:89 , srv1996367679:89 , srv307228853:89 , srv473943284:89 , srv65837124:89 , srv883653820:89 , srv118609237:90 , srv1488608033:90 , srv1897235971:90 , srv823931606:90 , srv274523191:91 , srv993829056:91 , srv1289306039:92 , srv1637439305:92 , srv358102847:95 , srv938651297:97 , srv1093683352:429 } 2023-05-12 19:53:23,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-12 19:53:23,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1403206514=20, srv274523191=57, srv700116442=85, srv2139226147=52, srv1897235971=39, srv792242379=90, srv118609237=9, srv531216229=77, srv1181549387=7, srv1289306039=14, srv2076031703=49, srv65837124=83, srv1016565859=0, srv1720307728=33, srv358102847=64, srv391763850=66, srv543492888=78, srv57703892=79, srv936548419=95, srv938651297=96, srv1168093835=5, srv3105223=60, srv913636735=93, srv1735017133=35, srv1194870565=11, srv1628128231=28, srv70105780=86, srv1613769265=27, srv1093683352=4, srv1819596117=36, srv2109871535=51, srv791745173=89, srv307228853=58, srv1693545208=31, srv344921439=62, srv1193576727=10, srv258434666=55, srv455078980=70, srv1935796059=40, srv2078967015=50, srv954907228=97, srv1946049863=41, srv1276959087=13, srv691507786=84, srv527055038=75, srv1547918752=26, srv227882650=53, srv204888190=47, srv42614937=67, srv1958400536=44, srv1724424879=34, srv372587919=65, srv1637439305=29, srv1714702987=32, srv993829056=99, srv139904411=18, srv489411269=74, srv1400803415=19, srv138639376=16, srv1459474246=23, srv1387433622=17, srv186663345=38, srv33600481=61, srv654579367=82, srv963597245=98, srv454925965=69, srv1018733457=1, srv530735550=76, srv768347900=88, srv1316050746=15, srv207376053=48, srv1506525301=25, srv1488608033=24, srv1692621681=30, srv1067036299=2, srv309715874=59, srv935887410=94, srv1082357286=3, srv883653820=92, srv1455968853=22, srv471362409=72, srv2005349747=46, srv619612979=80, srv463649486=71, srv1957365466=43, srv1996367679=45, srv244896692=54, srv823931606=91, srv1176282496=6, srv1950779215=42, srv1829450491=37, srv1181749762=8, srv348904254=63, srv259964897=56, srv642466468=81, srv1227909756=12, srv1428903602=21, srv473943284=73, srv716695538=87, srv441872125=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-12 19:53:23,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-12 19:53:23,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-12 19:53:23,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-12 19:53:23,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-12 19:53:23,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-12 19:53:23,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-12 19:53:23,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-12 19:53:23,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-12 19:53:23,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-12 19:53:23,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-12 19:53:23,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-12 19:53:23,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-12 19:53:23,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-12 19:53:23,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-12 19:53:23,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-12 19:53:23,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-12 19:53:23,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-12 19:53:23,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-12 19:53:23,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-12 19:53:23,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-12 19:53:23,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-12 19:53:23,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-12 19:53:23,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-12 19:53:23,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-12 19:53:23,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-12 19:53:23,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-12 19:53:23,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-12 19:53:23,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-12 19:53:23,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-12 19:53:23,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-12 19:53:23,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-12 19:53:23,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-12 19:53:23,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-12 19:53:23,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-12 19:53:23,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-12 19:53:23,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-12 19:53:23,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-12 19:53:23,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-12 19:53:23,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-12 19:53:23,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-12 19:53:23,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-12 19:53:23,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-12 19:53:23,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-12 19:53:23,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-12 19:53:23,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-12 19:53:23,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-12 19:53:23,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-12 19:53:23,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-12 19:53:23,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-12 19:53:23,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-12 19:53:23,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-12 19:53:23,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-12 19:53:23,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-12 19:53:23,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-12 19:53:23,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-12 19:53:23,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-12 19:53:23,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-12 19:53:23,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-12 19:53:23,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-12 19:53:23,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-12 19:53:23,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-12 19:53:23,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-12 19:53:23,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-12 19:53:23,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-12 19:53:23,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-12 19:53:23,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-12 19:53:23,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-12 19:53:23,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-12 19:53:23,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-12 19:53:23,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-12 19:53:23,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-12 19:53:23,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-12 19:53:23,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-12 19:53:23,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-12 19:53:23,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-12 19:53:23,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-12 19:53:23,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-12 19:53:23,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-12 19:53:23,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-12 19:53:23,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-12 19:53:23,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-12 19:53:23,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-12 19:53:23,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-12 19:53:23,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-12 19:53:23,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-12 19:53:23,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-12 19:53:23,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-12 19:53:23,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-12 19:53:23,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-12 19:53:23,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-12 19:53:23,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-12 19:53:23,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-12 19:53:23,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-12 19:53:23,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-12 19:53:23,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-12 19:53:23,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-12 19:53:23,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-12 19:53:23,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-12 19:53:23,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-12 19:53:23,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-12 19:53:23,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-12 19:53:23,574 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03815190723854308, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.058580285349679136, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03815190723854308, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17858333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24208333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-12 19:53:23,574 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-12 19:53:23,575 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18301599862900625, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03815190723854308, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.058580285349679136, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03815190723854308, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17858333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24208333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-12 19:53:23,886 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 466 ms to try 5000 different iterations. Found a solution that moves 2690 regions; Going from a computed imbalance of 0.18301599862900625 to a new imbalance of 0.014800257922306053. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019612787387464165); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034433038189614525, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29888888888888887, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019612787387464165); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04725, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-12 19:53:23,896 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv938651297:75 , srv1637439305:78 , srv274523191:79 , srv358102847:79 , srv823931606:80 , srv118609237:81 , srv1897235971:81 , srv993829056:81 , srv1289306039:82 , srv1455968853:82 , srv1488608033:82 , srv1724424879:82 , srv307228853:82 , srv57703892:82 , srv1428903602:83 , srv1935796059:83 , srv2005349747:83 , srv309715874:83 , srv473943284:83 , srv543492888:83 , srv65837124:83 , srv1082357286:84 , srv1829450491:84 , srv1996367679:84 , srv348904254:84 , srv471362409:84 , srv700116442:84 , srv913636735:84 , srv1018733457:85 , srv1168093835:85 , srv139904411:85 , srv1403206514:85 , srv1950779215:85 , srv530735550:85 , srv531216229:85 , srv619612979:85 , srv954907228:85 , srv1181549387:86 , srv1193576727:86 , srv1400803415:86 , srv1506525301:86 , srv1613769265:86 , srv1692621681:86 , srv1720307728:86 , srv186663345:86 , srv204888190:86 , srv2139226147:86 , srv259964897:86 , srv3105223:86 , srv489411269:86 , srv642466468:86 , srv691507786:86 , srv70105780:86 , srv791745173:86 , srv1176282496:87 , srv1227909756:87 , srv1459474246:87 , srv1547918752:87 , srv1628128231:87 , srv1819596117:87 , srv1957365466:87 , srv2076031703:87 , srv2078967015:87 , srv2109871535:87 , srv227882650:87 , srv258434666:87 , srv344921439:87 , srv372587919:87 , srv391763850:87 , srv654579367:87 , srv716695538:87 , srv792242379:87 , srv936548419:87 , srv963597245:87 , srv1016565859:88 , srv1194870565:88 , srv138639376:88 , srv1693545208:88 , srv1735017133:88 , srv1946049863:88 , srv1958400536:88 , srv244896692:88 , srv42614937:88 , srv441872125:88 , srv454925965:88 , srv455078980:88 , srv768347900:88 , srv1067036299:89 , srv1181749762:89 , srv1276959087:89 , srv1316050746:89 , srv1714702987:89 , srv207376053:89 , srv33600481:89 , srv883653820:89 , srv527055038:90 , srv1387433622:91 , srv463649486:91 , srv935887410:91 , srv1093683352:519 } 2023-05-12 19:53:23,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-12 19:53:23,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1403206514=20, srv274523191=57, srv700116442=85, srv2139226147=52, srv1897235971=39, srv792242379=90, srv118609237=9, srv531216229=77, srv1181549387=7, srv1289306039=14, srv2076031703=49, srv65837124=83, srv1016565859=0, srv1720307728=33, srv358102847=64, srv391763850=66, srv543492888=78, srv57703892=79, srv936548419=95, srv938651297=96, srv1168093835=5, srv3105223=60, srv913636735=93, srv1735017133=35, srv1194870565=11, srv1628128231=28, srv70105780=86, srv1613769265=27, srv1093683352=4, srv1819596117=36, srv2109871535=51, srv791745173=89, srv307228853=58, srv1693545208=31, srv344921439=62, srv1193576727=10, srv258434666=55, srv455078980=70, srv1935796059=40, srv2078967015=50, srv954907228=97, srv1946049863=41, srv1276959087=13, srv691507786=84, srv527055038=75, srv1547918752=26, srv227882650=53, srv204888190=47, srv42614937=67, srv1958400536=44, srv1724424879=34, srv372587919=65, srv1637439305=29, srv1714702987=32, srv993829056=99, srv139904411=18, srv489411269=74, srv1400803415=19, srv138639376=16, srv1459474246=23, srv1387433622=17, srv186663345=38, srv33600481=61, srv654579367=82, srv963597245=98, srv454925965=69, srv1018733457=1, srv530735550=76, srv768347900=88, srv1316050746=15, srv207376053=48, srv1506525301=25, srv1488608033=24, srv1692621681=30, srv1067036299=2, srv309715874=59, srv935887410=94, srv1082357286=3, srv883653820=92, srv1455968853=22, srv471362409=72, srv2005349747=46, srv619612979=80, srv463649486=71, srv1957365466=43, srv1996367679=45, srv244896692=54, srv823931606=91, srv1176282496=6, srv1950779215=42, srv1829450491=37, srv1181749762=8, srv348904254=63, srv259964897=56, srv642466468=81, srv1227909756=12, srv1428903602=21, srv473943284=73, srv716695538=87, srv441872125=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-12 19:53:23,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-12 19:53:23,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-12 19:53:23,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-12 19:53:23,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-12 19:53:23,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-12 19:53:23,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-12 19:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-12 19:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-12 19:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-12 19:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-12 19:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-12 19:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-12 19:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-12 19:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-12 19:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-12 19:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-12 19:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-12 19:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-12 19:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-12 19:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-12 19:53:23,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-12 19:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-12 19:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-12 19:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-12 19:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-12 19:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-12 19:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-12 19:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-12 19:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-12 19:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-12 19:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-12 19:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-12 19:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-12 19:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-12 19:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-12 19:53:23,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-12 19:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-12 19:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-12 19:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-12 19:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-12 19:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-12 19:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-12 19:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-12 19:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-12 19:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-12 19:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-12 19:53:23,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-12 19:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-12 19:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-12 19:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-12 19:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-12 19:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-12 19:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-12 19:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-12 19:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-12 19:53:23,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-12 19:53:23,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-12 19:53:23,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-12 19:53:23,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-12 19:53:23,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-12 19:53:23,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-12 19:53:23,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-12 19:53:23,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-12 19:53:23,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-12 19:53:23,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-12 19:53:23,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-12 19:53:23,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-12 19:53:23,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-12 19:53:23,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-12 19:53:23,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-12 19:53:23,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-12 19:53:23,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-12 19:53:23,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-12 19:53:23,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-12 19:53:23,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-12 19:53:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-12 19:53:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-12 19:53:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-12 19:53:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-12 19:53:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-12 19:53:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-12 19:53:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-12 19:53:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-12 19:53:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-12 19:53:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-12 19:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-12 19:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-12 19:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-12 19:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-12 19:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-12 19:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-12 19:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-12 19:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-12 19:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-12 19:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-12 19:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-12 19:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-12 19:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-12 19:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-12 19:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-12 19:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-12 19:53:24,076 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019612787387464165); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034433038189614525, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019612787387464165); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04725, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-12 19:53:24,076 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-12 19:53:24,077 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0147814221397208, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019612787387464165); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034433038189614525, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019612787387464165); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04725, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-12 19:53:24,413 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 469 ms to try 5000 different iterations. Found a solution that moves 990 regions; Going from a computed imbalance of 0.0147814221397208 to a new imbalance of 6.566994298465933E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0045360921162651445); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007828119907069723); MoveCostFunction : (multiplier=7.0, imbalance=0.11, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0045360921162651445); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-12 19:53:24,415 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv463649486:78 , srv883653820:80 , srv1714702987:82 , srv207376053:82 , srv935887410:82 , srv1181549387:83 , srv1276959087:83 , srv1289306039:83 , srv1387433622:83 , srv1819596117:83 , srv2109871535:83 , srv792242379:83 , srv1316050746:84 , srv138639376:84 , srv1400803415:84 , srv1628128231:84 , srv1693545208:84 , srv186663345:84 , srv1950779215:84 , srv1957365466:84 , srv259964897:84 , srv33600481:84 , srv358102847:84 , srv372587919:84 , srv42614937:84 , srv619612979:84 , srv70105780:84 , srv768347900:84 , srv936548419:84 , srv963597245:84 , srv1067036299:85 , srv1168093835:85 , srv1176282496:85 , srv1181749762:85 , srv1194870565:85 , srv1227909756:85 , srv1613769265:85 , srv1958400536:85 , srv1996367679:85 , srv2076031703:85 , srv258434666:85 , srv3105223:85 , srv455078980:85 , srv489411269:85 , srv527055038:85 , srv654579367:85 , srv716695538:85 , srv1082357286:86 , srv118609237:86 , srv1459474246:86 , srv1488608033:86 , srv1692621681:86 , srv1720307728:86 , srv1724424879:86 , srv1829450491:86 , srv1897235971:86 , srv2005349747:86 , srv204888190:86 , srv2078967015:86 , srv2139226147:86 , srv244896692:86 , srv307228853:86 , srv348904254:86 , srv391763850:86 , srv473943284:86 , srv530735550:86 , srv543492888:86 , srv642466468:86 , srv65837124:86 , srv691507786:86 , srv700116442:86 , srv791745173:86 , srv938651297:86 , srv1016565859:87 , srv1018733457:87 , srv1193576727:87 , srv139904411:87 , srv1403206514:87 , srv1428903602:87 , srv1506525301:87 , srv1547918752:87 , srv1735017133:87 , srv227882650:87 , srv344921439:87 , srv441872125:87 , srv471362409:87 , srv823931606:87 , srv913636735:87 , srv954907228:87 , srv993829056:87 , srv1455968853:88 , srv1637439305:88 , srv1935796059:88 , srv274523191:88 , srv309715874:88 , srv454925965:88 , srv531216229:88 , srv57703892:88 , srv1946049863:89 , srv1093683352:550 } 2023-05-12 19:53:24,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-12 19:53:24,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1403206514=20, srv274523191=57, srv700116442=85, srv2139226147=52, srv1897235971=39, srv792242379=90, srv118609237=9, srv531216229=77, srv1181549387=7, srv1289306039=14, srv2076031703=49, srv65837124=83, srv1016565859=0, srv1720307728=33, srv358102847=64, srv391763850=66, srv543492888=78, srv57703892=79, srv936548419=95, srv938651297=96, srv1168093835=5, srv3105223=60, srv913636735=93, srv1735017133=35, srv1194870565=11, srv1628128231=28, srv70105780=86, srv1613769265=27, srv1093683352=4, srv1819596117=36, srv2109871535=51, srv791745173=89, srv307228853=58, srv1693545208=31, srv344921439=62, srv1193576727=10, srv258434666=55, srv455078980=70, srv1935796059=40, srv2078967015=50, srv954907228=97, srv1946049863=41, srv1276959087=13, srv691507786=84, srv527055038=75, srv1547918752=26, srv227882650=53, srv204888190=47, srv42614937=67, srv1958400536=44, srv1724424879=34, srv372587919=65, srv1637439305=29, srv1714702987=32, srv993829056=99, srv139904411=18, srv489411269=74, srv1400803415=19, srv138639376=16, srv1459474246=23, srv1387433622=17, srv186663345=38, srv33600481=61, srv654579367=82, srv963597245=98, srv454925965=69, srv1018733457=1, srv530735550=76, srv768347900=88, srv1316050746=15, srv207376053=48, srv1506525301=25, srv1488608033=24, srv1692621681=30, srv1067036299=2, srv309715874=59, srv935887410=94, srv1082357286=3, srv883653820=92, srv1455968853=22, srv471362409=72, srv2005349747=46, srv619612979=80, srv463649486=71, srv1957365466=43, srv1996367679=45, srv244896692=54, srv823931606=91, srv1176282496=6, srv1950779215=42, srv1829450491=37, srv1181749762=8, srv348904254=63, srv259964897=56, srv642466468=81, srv1227909756=12, srv1428903602=21, srv473943284=73, srv716695538=87, srv441872125=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-12 19:53:24,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-12 19:53:24,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-12 19:53:24,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-12 19:53:24,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-12 19:53:24,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-12 19:53:24,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-12 19:53:24,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-12 19:53:24,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-12 19:53:24,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-12 19:53:24,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-12 19:53:24,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-12 19:53:24,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-12 19:53:24,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-12 19:53:24,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-12 19:53:24,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-12 19:53:24,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-12 19:53:24,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-12 19:53:24,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-12 19:53:24,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-12 19:53:24,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-12 19:53:24,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-12 19:53:24,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-12 19:53:24,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-12 19:53:24,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-12 19:53:24,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-12 19:53:24,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-12 19:53:24,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-12 19:53:24,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-12 19:53:24,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-12 19:53:24,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-12 19:53:24,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-12 19:53:24,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-12 19:53:24,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-12 19:53:24,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-12 19:53:24,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-12 19:53:24,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-12 19:53:24,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-12 19:53:24,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-12 19:53:24,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-12 19:53:24,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-12 19:53:24,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-12 19:53:24,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-12 19:53:24,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-12 19:53:24,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-12 19:53:24,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-12 19:53:24,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-12 19:53:24,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-12 19:53:24,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-12 19:53:24,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-12 19:53:24,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-12 19:53:24,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-12 19:53:24,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-12 19:53:24,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-12 19:53:24,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-12 19:53:24,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-12 19:53:24,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-12 19:53:24,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-12 19:53:24,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-12 19:53:24,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-12 19:53:24,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-12 19:53:24,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-12 19:53:24,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-12 19:53:24,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-12 19:53:24,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-12 19:53:24,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-12 19:53:24,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-12 19:53:24,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-12 19:53:24,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-12 19:53:24,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-12 19:53:24,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-12 19:53:24,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-12 19:53:24,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-12 19:53:24,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-12 19:53:24,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-12 19:53:24,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-12 19:53:24,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-12 19:53:24,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-12 19:53:24,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-12 19:53:24,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-12 19:53:24,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-12 19:53:24,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-12 19:53:24,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-12 19:53:24,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-12 19:53:24,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-12 19:53:24,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-12 19:53:24,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-12 19:53:24,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-12 19:53:24,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-12 19:53:24,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-12 19:53:24,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-12 19:53:24,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-12 19:53:24,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-12 19:53:24,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-12 19:53:24,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-12 19:53:24,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-12 19:53:24,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-12 19:53:24,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-12 19:53:24,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-12 19:53:24,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-12 19:53:24,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-12 19:53:24,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-12 19:53:24,563 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0045360921162651445); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007828119907069723); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0045360921162651445); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-12 19:53:24,564 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-12 19:53:24,564 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.497673016832472E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0045360921162651445); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007828119907069723); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0045360921162651445); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-12 19:53:24,969 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 508 ms to try 5000 different iterations. Found a solution that moves 261 regions; Going from a computed imbalance of 6.497673016832472E-4 to a new imbalance of 2.7644763261550742E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.029, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-12 19:53:24,971 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1227909756:83 , srv1935796059:83 , srv1946049863:83 , srv274523191:83 , srv309715874:83 , srv3105223:83 , srv993829056:83 , srv1016565859:84 , srv1018733457:84 , srv1168093835:84 , srv1181749762:84 , srv1193576727:84 , srv1276959087:84 , srv139904411:84 , srv1400803415:84 , srv1403206514:84 , srv1428903602:84 , srv1488608033:84 , srv1547918752:84 , srv1613769265:84 , srv1637439305:84 , srv1692621681:84 , srv1714702987:84 , srv1735017133:84 , srv1819596117:84 , srv1950779215:84 , srv1957365466:84 , srv1958400536:84 , srv1996367679:84 , srv204888190:84 , srv207376053:84 , srv227882650:84 , srv244896692:84 , srv259964897:84 , srv344921439:84 , srv348904254:84 , srv358102847:84 , srv372587919:84 , srv441872125:84 , srv455078980:84 , srv471362409:84 , srv527055038:84 , srv530735550:84 , srv531216229:84 , srv57703892:84 , srv642466468:84 , srv65837124:84 , srv700116442:84 , srv791745173:84 , srv883653820:84 , srv913636735:84 , srv938651297:84 , srv954907228:84 , srv963597245:84 , srv1067036299:85 , srv1082357286:85 , srv1176282496:85 , srv1181549387:85 , srv118609237:85 , srv1194870565:85 , srv1289306039:85 , srv1316050746:85 , srv138639376:85 , srv1455968853:85 , srv1459474246:85 , srv1506525301:85 , srv1628128231:85 , srv1693545208:85 , srv1720307728:85 , srv1724424879:85 , srv1829450491:85 , srv186663345:85 , srv1897235971:85 , srv2005349747:85 , srv2076031703:85 , srv2078967015:85 , srv2139226147:85 , srv258434666:85 , srv307228853:85 , srv33600481:85 , srv391763850:85 , srv42614937:85 , srv454925965:85 , srv463649486:85 , srv473943284:85 , srv489411269:85 , srv543492888:85 , srv619612979:85 , srv654579367:85 , srv691507786:85 , srv70105780:85 , srv716695538:85 , srv768347900:85 , srv792242379:85 , srv823931606:85 , srv935887410:85 , srv936548419:85 , srv1387433622:86 , srv2109871535:86 , srv1093683352:644 } 2023-05-12 19:53:24,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-12 19:53:24,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1403206514=20, srv274523191=57, srv700116442=85, srv2139226147=52, srv1897235971=39, srv792242379=90, srv118609237=9, srv531216229=77, srv1181549387=7, srv1289306039=14, srv2076031703=49, srv65837124=83, srv1016565859=0, srv1720307728=33, srv358102847=64, srv391763850=66, srv543492888=78, srv57703892=79, srv936548419=95, srv938651297=96, srv1168093835=5, srv3105223=60, srv913636735=93, srv1735017133=35, srv1194870565=11, srv1628128231=28, srv70105780=86, srv1613769265=27, srv1093683352=4, srv1819596117=36, srv2109871535=51, srv791745173=89, srv307228853=58, srv1693545208=31, srv344921439=62, srv1193576727=10, srv258434666=55, srv455078980=70, srv1935796059=40, srv2078967015=50, srv954907228=97, srv1946049863=41, srv1276959087=13, srv691507786=84, srv527055038=75, srv1547918752=26, srv227882650=53, srv204888190=47, srv42614937=67, srv1958400536=44, srv1724424879=34, srv372587919=65, srv1637439305=29, srv1714702987=32, srv993829056=99, srv139904411=18, srv489411269=74, srv1400803415=19, srv138639376=16, srv1459474246=23, srv1387433622=17, srv186663345=38, srv33600481=61, srv654579367=82, srv963597245=98, srv454925965=69, srv1018733457=1, srv530735550=76, srv768347900=88, srv1316050746=15, srv207376053=48, srv1506525301=25, srv1488608033=24, srv1692621681=30, srv1067036299=2, srv309715874=59, srv935887410=94, srv1082357286=3, srv883653820=92, srv1455968853=22, srv471362409=72, srv2005349747=46, srv619612979=80, srv463649486=71, srv1957365466=43, srv1996367679=45, srv244896692=54, srv823931606=91, srv1176282496=6, srv1950779215=42, srv1829450491=37, srv1181749762=8, srv348904254=63, srv259964897=56, srv642466468=81, srv1227909756=12, srv1428903602=21, srv473943284=73, srv716695538=87, srv441872125=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-12 19:53:24,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-12 19:53:24,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-12 19:53:24,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-12 19:53:24,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-12 19:53:24,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-12 19:53:24,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-12 19:53:24,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-12 19:53:24,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-12 19:53:24,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-12 19:53:24,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-12 19:53:24,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-12 19:53:24,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-12 19:53:24,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-12 19:53:24,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-12 19:53:24,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-12 19:53:24,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-12 19:53:24,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-12 19:53:24,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-12 19:53:24,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-12 19:53:24,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-12 19:53:24,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-12 19:53:24,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-12 19:53:24,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-12 19:53:24,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-12 19:53:24,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-12 19:53:24,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-12 19:53:24,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-12 19:53:24,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-12 19:53:24,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-12 19:53:24,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-12 19:53:24,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-12 19:53:24,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-12 19:53:24,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-12 19:53:24,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-12 19:53:24,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-12 19:53:24,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-12 19:53:24,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-12 19:53:24,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-12 19:53:24,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-12 19:53:24,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-12 19:53:24,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-12 19:53:24,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-12 19:53:24,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-12 19:53:24,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-12 19:53:24,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-12 19:53:24,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-12 19:53:24,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-12 19:53:24,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-12 19:53:24,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-12 19:53:24,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-12 19:53:24,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-12 19:53:24,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-12 19:53:24,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-12 19:53:24,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-12 19:53:24,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-12 19:53:24,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-12 19:53:24,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-12 19:53:24,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-12 19:53:24,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-12 19:53:24,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-12 19:53:24,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-12 19:53:24,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-12 19:53:24,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-12 19:53:24,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-12 19:53:24,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-12 19:53:24,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-12 19:53:24,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-12 19:53:24,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-12 19:53:24,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-12 19:53:24,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-12 19:53:24,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-12 19:53:24,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-12 19:53:24,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-12 19:53:24,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-12 19:53:24,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-12 19:53:24,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-12 19:53:24,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-12 19:53:24,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-12 19:53:24,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-12 19:53:24,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-12 19:53:24,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-12 19:53:24,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-12 19:53:24,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-12 19:53:24,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-12 19:53:24,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-12 19:53:24,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-12 19:53:24,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-12 19:53:24,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-12 19:53:24,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-12 19:53:24,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-12 19:53:24,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-12 19:53:24,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-12 19:53:24,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-12 19:53:24,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-12 19:53:24,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-12 19:53:24,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-12 19:53:24,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-12 19:53:24,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-12 19:53:24,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-12 19:53:24,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-12 19:53:24,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-12 19:53:25,131 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-12 19:53:25,132 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-12 19:53:25,132 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.581720220030494E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-12 19:53:25,402 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 393 ms to try 5000 different iterations. Found a solution that moves 31 regions; Going from a computed imbalance of 2.581720220030494E-5 to a new imbalance of 6.615052311728012E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0034444444444444444); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-12 19:53:25,403 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1316050746:83 , srv1387433622:83 , srv1455968853:83 , srv1720307728:83 , srv2078967015:83 , srv33600481:83 , srv391763850:83 , srv454925965:83 , srv473943284:83 , srv1016565859:84 , srv1018733457:84 , srv1067036299:84 , srv1082357286:84 , srv1168093835:84 , srv1176282496:84 , srv1181549387:84 , srv118609237:84 , srv1193576727:84 , srv1194870565:84 , srv1276959087:84 , srv1289306039:84 , srv138639376:84 , srv139904411:84 , srv1403206514:84 , srv1428903602:84 , srv1459474246:84 , srv1488608033:84 , srv1506525301:84 , srv1547918752:84 , srv1613769265:84 , srv1628128231:84 , srv1637439305:84 , srv1693545208:84 , srv1714702987:84 , srv1724424879:84 , srv1735017133:84 , srv1829450491:84 , srv186663345:84 , srv1897235971:84 , srv1935796059:84 , srv1946049863:84 , srv1957365466:84 , srv1958400536:84 , srv1996367679:84 , srv2005349747:84 , srv204888190:84 , srv207376053:84 , srv2076031703:84 , srv2109871535:84 , srv2139226147:84 , srv227882650:84 , srv244896692:84 , srv258434666:84 , srv259964897:84 , srv307228853:84 , srv309715874:84 , srv344921439:84 , srv348904254:84 , srv358102847:84 , srv372587919:84 , srv42614937:84 , srv455078980:84 , srv463649486:84 , srv471362409:84 , srv489411269:84 , srv527055038:84 , srv530735550:84 , srv531216229:84 , srv543492888:84 , srv57703892:84 , srv619612979:84 , srv642466468:84 , srv654579367:84 , srv65837124:84 , srv691507786:84 , srv700116442:84 , srv70105780:84 , srv716695538:84 , srv768347900:84 , srv791745173:84 , srv792242379:84 , srv823931606:84 , srv883653820:84 , srv913636735:84 , srv935887410:84 , srv936548419:84 , srv938651297:84 , srv954907228:84 , srv963597245:84 , srv993829056:84 , srv1181749762:85 , srv1227909756:85 , srv1400803415:85 , srv1692621681:85 , srv1819596117:85 , srv1950779215:85 , srv274523191:85 , srv3105223:85 , srv441872125:85 , srv1093683352:684 } 2023-05-12 19:53:25,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-12 19:53:25,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1403206514=20, srv274523191=57, srv700116442=85, srv2139226147=52, srv1897235971=39, srv792242379=90, srv118609237=9, srv531216229=77, srv1181549387=7, srv1289306039=14, srv2076031703=49, srv65837124=83, srv1016565859=0, srv1720307728=33, srv358102847=64, srv391763850=66, srv543492888=78, srv57703892=79, srv936548419=95, srv938651297=96, srv1168093835=5, srv3105223=60, srv913636735=93, srv1735017133=35, srv1194870565=11, srv1628128231=28, srv70105780=86, srv1613769265=27, srv1093683352=4, srv1819596117=36, srv2109871535=51, srv791745173=89, srv307228853=58, srv1693545208=31, srv344921439=62, srv1193576727=10, srv258434666=55, srv455078980=70, srv1935796059=40, srv2078967015=50, srv954907228=97, srv1946049863=41, srv1276959087=13, srv691507786=84, srv527055038=75, srv1547918752=26, srv227882650=53, srv204888190=47, srv42614937=67, srv1958400536=44, srv1724424879=34, srv372587919=65, srv1637439305=29, srv1714702987=32, srv993829056=99, srv139904411=18, srv489411269=74, srv1400803415=19, srv138639376=16, srv1459474246=23, srv1387433622=17, srv186663345=38, srv33600481=61, srv654579367=82, srv963597245=98, srv454925965=69, srv1018733457=1, srv530735550=76, srv768347900=88, srv1316050746=15, srv207376053=48, srv1506525301=25, srv1488608033=24, srv1692621681=30, srv1067036299=2, srv309715874=59, srv935887410=94, srv1082357286=3, srv883653820=92, srv1455968853=22, srv471362409=72, srv2005349747=46, srv619612979=80, srv463649486=71, srv1957365466=43, srv1996367679=45, srv244896692=54, srv823931606=91, srv1176282496=6, srv1950779215=42, srv1829450491=37, srv1181749762=8, srv348904254=63, srv259964897=56, srv642466468=81, srv1227909756=12, srv1428903602=21, srv473943284=73, srv716695538=87, srv441872125=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-12 19:53:25,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-12 19:53:25,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-12 19:53:25,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-12 19:53:25,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-12 19:53:25,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-12 19:53:25,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-12 19:53:25,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-12 19:53:25,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-12 19:53:25,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-12 19:53:25,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-12 19:53:25,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-12 19:53:25,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-12 19:53:25,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-12 19:53:25,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-12 19:53:25,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-12 19:53:25,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-12 19:53:25,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-12 19:53:25,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-12 19:53:25,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-12 19:53:25,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-12 19:53:25,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-12 19:53:25,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-12 19:53:25,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-12 19:53:25,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-12 19:53:25,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-12 19:53:25,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-12 19:53:25,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-12 19:53:25,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-12 19:53:25,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-12 19:53:25,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-12 19:53:25,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-12 19:53:25,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-12 19:53:25,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-12 19:53:25,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-12 19:53:25,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-12 19:53:25,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-12 19:53:25,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-12 19:53:25,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-12 19:53:25,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-12 19:53:25,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-12 19:53:25,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-12 19:53:25,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-12 19:53:25,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-12 19:53:25,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-12 19:53:25,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-12 19:53:25,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-12 19:53:25,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-12 19:53:25,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-12 19:53:25,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-12 19:53:25,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-12 19:53:25,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-12 19:53:25,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-12 19:53:25,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-12 19:53:25,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-12 19:53:25,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-12 19:53:25,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-12 19:53:25,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-12 19:53:25,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-12 19:53:25,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-12 19:53:25,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-12 19:53:25,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-12 19:53:25,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-12 19:53:25,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-12 19:53:25,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-12 19:53:25,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-12 19:53:25,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-12 19:53:25,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-12 19:53:25,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-12 19:53:25,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-12 19:53:25,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-12 19:53:25,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-12 19:53:25,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-12 19:53:25,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-12 19:53:25,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-12 19:53:25,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-12 19:53:25,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-12 19:53:25,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-12 19:53:25,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-12 19:53:25,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-12 19:53:25,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-12 19:53:25,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-12 19:53:25,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-12 19:53:25,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-12 19:53:25,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-12 19:53:25,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-12 19:53:25,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-12 19:53:25,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-12 19:53:25,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-12 19:53:25,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-12 19:53:25,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-12 19:53:25,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-12 19:53:25,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-12 19:53:25,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-12 19:53:25,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-12 19:53:25,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-12 19:53:25,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-12 19:53:25,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-12 19:53:25,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-12 19:53:25,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-12 19:53:25,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-12 19:53:25,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-12 19:53:25,538 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.397985672269699E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-12 19:53:25,538 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1316050746:83 , srv1387433622:83 , srv1455968853:83 , srv1720307728:83 , srv2078967015:83 , srv33600481:83 , srv391763850:83 , srv454925965:83 , srv473943284:83 , srv1016565859:84 , srv1018733457:84 , srv1067036299:84 , srv1082357286:84 , srv1168093835:84 , srv1176282496:84 , srv1181549387:84 , srv118609237:84 , srv1193576727:84 , srv1194870565:84 , srv1276959087:84 , srv1289306039:84 , srv138639376:84 , srv139904411:84 , srv1403206514:84 , srv1428903602:84 , srv1459474246:84 , srv1488608033:84 , srv1506525301:84 , srv1547918752:84 , srv1613769265:84 , srv1628128231:84 , srv1637439305:84 , srv1693545208:84 , srv1714702987:84 , srv1724424879:84 , srv1735017133:84 , srv1829450491:84 , srv186663345:84 , srv1897235971:84 , srv1935796059:84 , srv1946049863:84 , srv1957365466:84 , srv1958400536:84 , srv1996367679:84 , srv2005349747:84 , srv204888190:84 , srv207376053:84 , srv2076031703:84 , srv2109871535:84 , srv2139226147:84 , srv227882650:84 , srv244896692:84 , srv258434666:84 , srv259964897:84 , srv307228853:84 , srv309715874:84 , srv344921439:84 , srv348904254:84 , srv358102847:84 , srv372587919:84 , srv42614937:84 , srv455078980:84 , srv463649486:84 , srv471362409:84 , srv489411269:84 , srv527055038:84 , srv530735550:84 , srv531216229:84 , srv543492888:84 , srv57703892:84 , srv619612979:84 , srv642466468:84 , srv654579367:84 , srv65837124:84 , srv691507786:84 , srv700116442:84 , srv70105780:84 , srv716695538:84 , srv768347900:84 , srv791745173:84 , srv792242379:84 , srv823931606:84 , srv883653820:84 , srv913636735:84 , srv935887410:84 , srv936548419:84 , srv938651297:84 , srv954907228:84 , srv963597245:84 , srv993829056:84 , srv1181749762:85 , srv1227909756:85 , srv1400803415:85 , srv1692621681:85 , srv1819596117:85 , srv1950779215:85 , srv274523191:85 , srv3105223:85 , srv441872125:85 , srv1093683352:684 } 2023-05-12 19:53:25,568 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=312 (was 312), ProcessCount=175 (was 175), AvailableMemoryMB=9506 (was 10391) 2023-05-12 19:53:25,576 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=312, ProcessCount=175, AvailableMemoryMB=9504 2023-05-12 19:53:25,577 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-12 19:53:25,577 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-05-12 19:53:25,589 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1055655141:84 , srv1130205011:84 , srv1152397779:84 , srv122477071:84 , srv124908309:84 , srv1284437670:84 , srv1362010221:84 , srv1465229466:84 , srv1591825891:84 , srv1610115578:84 , srv1741096825:84 , srv1803838813:84 , srv1855252066:84 , srv1856486979:84 , srv2041908148:84 , srv2062138196:84 , srv2091426174:84 , srv2096327384:84 , srv2117523417:84 , srv2121128555:84 , srv428901476:84 , srv432746581:84 , srv500770449:84 , srv52711787:84 , srv73580231:84 , srv810364713:84 , srv91634402:84 , srv947733466:84 , srv996243430:84 , srv243744966:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-05-12 19:53:25,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-12 19:53:25,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv73580231=25, srv428901476=21, srv500770449=23, srv1803838813=11, srv996243430=29, srv1130205011=1, srv1591825891=8, srv1610115578=9, srv947733466=28, srv432746581=22, srv1465229466=7, srv810364713=26, srv2121128555=19, srv2062138196=15, srv2041908148=14, srv52711787=24, srv1856486979=13, srv2096327384=17, srv1362010221=6, srv2117523417=18, srv124908309=4, srv1741096825=10, srv91634402=27, srv1284437670=5, srv122477071=3, srv1055655141=0, srv1855252066=12, srv2091426174=16, srv243744966=20, srv1152397779=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-12 19:53:25,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-12 19:53:25,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-12 19:53:25,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-12 19:53:25,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-12 19:53:25,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-12 19:53:25,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-12 19:53:25,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-12 19:53:25,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-12 19:53:25,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-12 19:53:25,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-12 19:53:25,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-12 19:53:25,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-12 19:53:25,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-12 19:53:25,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-12 19:53:25,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-12 19:53:25,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-12 19:53:25,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-12 19:53:25,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-12 19:53:25,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-12 19:53:25,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-12 19:53:25,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-12 19:53:25,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-12 19:53:25,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-12 19:53:25,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-12 19:53:25,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-12 19:53:25,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-12 19:53:25,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-12 19:53:25,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-12 19:53:25,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-12 19:53:25,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-12 19:53:25,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-12 19:53:25,626 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.163091749183892, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-12 19:53:25,626 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912706940040521, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.163091749183892, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-05-12 19:55:25,608 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6487897 different iterations. Found a solution that moves 1859 regions; Going from a computed imbalance of 0.9912706940040521 to a new imbalance of 4.338998739279625E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6885185185185185, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-12 19:55:25,615 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1055655141:90 , srv1130205011:90 , srv1152397779:90 , srv122477071:90 , srv124908309:90 , srv1284437670:90 , srv1362010221:90 , srv1465229466:90 , srv1591825891:90 , srv1610115578:90 , srv1741096825:90 , srv1803838813:90 , srv1855252066:90 , srv1856486979:90 , srv2041908148:90 , srv2062138196:90 , srv2091426174:90 , srv2096327384:90 , srv2117523417:90 , srv2121128555:90 , srv243744966:90 , srv428901476:90 , srv432746581:90 , srv500770449:90 , srv52711787:90 , srv73580231:90 , srv810364713:90 , srv91634402:90 , srv947733466:90 , srv996243430:90 } 2023-05-12 19:55:25,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-12 19:55:25,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv73580231=25, srv428901476=21, srv500770449=23, srv1803838813=11, srv996243430=29, srv1130205011=1, srv1591825891=8, srv1610115578=9, srv947733466=28, srv432746581=22, srv1465229466=7, srv810364713=26, srv2121128555=19, srv2062138196=15, srv2041908148=14, srv52711787=24, srv1856486979=13, srv2096327384=17, srv1362010221=6, srv2117523417=18, srv124908309=4, srv1741096825=10, srv91634402=27, srv1284437670=5, srv122477071=3, srv1055655141=0, srv1855252066=12, srv2091426174=16, srv243744966=20, srv1152397779=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-12 19:55:25,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-12 19:55:25,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-12 19:55:25,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-12 19:55:25,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-12 19:55:25,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-12 19:55:25,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-12 19:55:25,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-12 19:55:25,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-12 19:55:25,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-12 19:55:25,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-12 19:55:25,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-12 19:55:25,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-12 19:55:25,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-12 19:55:25,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-12 19:55:25,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-12 19:55:25,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-12 19:55:25,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-12 19:55:25,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-12 19:55:25,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-12 19:55:25,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-12 19:55:25,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-12 19:55:25,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-12 19:55:25,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-12 19:55:25,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-12 19:55:25,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-12 19:55:25,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-12 19:55:25,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-12 19:55:25,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-12 19:55:25,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-12 19:55:25,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-12 19:55:25,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-12 19:55:25,634 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-12 19:55:25,634 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1055655141:90 , srv1130205011:90 , srv1152397779:90 , srv122477071:90 , srv124908309:90 , srv1284437670:90 , srv1362010221:90 , srv1465229466:90 , srv1591825891:90 , srv1610115578:90 , srv1741096825:90 , srv1803838813:90 , srv1855252066:90 , srv1856486979:90 , srv2041908148:90 , srv2062138196:90 , srv2091426174:90 , srv2096327384:90 , srv2117523417:90 , srv2121128555:90 , srv243744966:90 , srv428901476:90 , srv432746581:90 , srv500770449:90 , srv52711787:90 , srv73580231:90 , srv810364713:90 , srv91634402:90 , srv947733466:90 , srv996243430:90 } 2023-05-12 19:55:25,647 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=171 (was 312), ProcessCount=170 (was 175), AvailableMemoryMB=9474 (was 9504)