2023-10-19 03:53:28,654 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-10-19 03:53:28,956 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-19 03:53:29,077 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-10-19 03:53:29,141 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-19 03:53:29,149 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-19 03:53:29,162 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=327, ProcessCount=169, AvailableMemoryMB=11356 2023-10-19 03:53:29,164 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-19 03:53:29,165 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-19 03:53:29,282 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1000418383:84 , srv1016625663:84 , srv1065597400:84 , srv1066067229:84 , srv1070484404:84 , srv1125492022:84 , srv1132468577:84 , srv1157733758:84 , srv1173590366:84 , srv1180196123:84 , srv1211798173:84 , srv1222259493:84 , srv1225832885:84 , srv1270964540:84 , srv1271917480:84 , srv1273436778:84 , srv1277088888:84 , srv1282259358:84 , srv1291738677:84 , srv1291852958:84 , srv1300718705:84 , srv1310955516:84 , srv1314468690:84 , srv1341729207:84 , srv1344116240:84 , srv1362952210:84 , srv1365766994:84 , srv1388533565:84 , srv1425799447:84 , srv1435815688:84 , srv1439809976:84 , srv1473159306:84 , srv1479791780:84 , srv1506641423:84 , srv1513039942:84 , srv1519150796:84 , srv1545853197:84 , srv154879625:84 , srv154892824:84 , srv1601576907:84 , srv1618064329:84 , srv1678257363:84 , srv1684459445:84 , srv1707251103:84 , srv1771252934:84 , srv1832548070:84 , srv1888906448:84 , srv1891843772:84 , srv1902894725:84 , srv1903772771:84 , srv1906940622:84 , srv1910230112:84 , srv1969881595:84 , srv198647643:84 , srv2012890043:84 , srv2026177989:84 , srv2059126585:84 , srv2070835034:84 , srv2082661444:84 , srv2097244291:84 , srv2101933639:84 , srv2109222463:84 , srv2117462791:84 , srv2123697544:84 , srv304488573:84 , srv309079506:84 , srv331016813:84 , srv379082845:84 , srv38481908:84 , srv399301600:84 , srv407592735:84 , srv427497669:84 , srv433015200:84 , srv446868888:84 , srv465588354:84 , srv471542568:84 , srv486926314:84 , srv507307106:84 , srv575910455:84 , srv588445009:84 , srv589492453:84 , srv609055584:84 , srv610608521:84 , srv624022252:84 , srv6482941:84 , srv662770599:84 , srv666604760:84 , srv679942789:84 , srv72469254:84 , srv727377218:84 , srv789579626:84 , srv802415169:84 , srv804104983:84 , srv812740927:84 , srv859413696:84 , srv894909562:84 , srv9028390:84 , srv963418293:84 , srv97720683:84 , srv450351636:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-10-19 03:53:29,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 03:53:29,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1545853197=36, srv2109222463=61, srv1425799447=28, srv1969881595=52, srv894909562=96, srv1065597400=2, srv1271917480=14, srv963418293=98, srv2123697544=63, srv379082845=67, srv450351636=74, srv2097244291=59, srv331016813=66, srv1618064329=40, srv1273436778=15, srv1678257363=41, srv1707251103=43, srv1066067229=3, srv1435815688=29, srv407592735=70, srv1479791780=32, srv304488573=64, srv97720683=99, srv1157733758=7, srv507307106=78, srv1506641423=33, srv1362952210=25, srv446868888=73, srv38481908=68, srv2059126585=56, srv427497669=71, srv1903772771=49, srv859413696=95, srv198647643=53, srv1180196123=9, srv1270964540=13, srv1300718705=20, srv1291738677=18, srv789579626=91, srv2082661444=58, srv1225832885=12, srv1125492022=5, srv588445009=80, srv1016625663=1, srv1513039942=34, srv1365766994=26, srv486926314=77, srv1291852958=19, srv1310955516=21, srv727377218=90, srv433015200=72, srv1771252934=44, srv2117462791=62, srv1282259358=17, srv1891843772=47, srv1388533565=27, srv1684459445=42, srv2026177989=55, srv1519150796=35, srv2101933639=60, srv610608521=83, srv1173590366=8, srv309079506=65, srv624022252=84, srv72469254=89, srv154879625=37, srv465588354=75, srv399301600=69, srv1000418383=0, srv1132468577=6, srv812740927=94, srv1888906448=46, srv9028390=97, srv2012890043=54, srv1902894725=48, srv1473159306=31, srv154892824=38, srv1341729207=23, srv1222259493=11, srv1910230112=51, srv1832548070=45, srv804104983=93, srv1344116240=24, srv1070484404=4, srv802415169=92, srv2070835034=57, srv6482941=85, srv609055584=82, srv589492453=81, srv471542568=76, srv679942789=88, srv1906940622=50, srv1314468690=22, srv1601576907=39, srv1439809976=30, srv666604760=87, srv575910455=79, srv1277088888=16, srv662770599=86, srv1211798173=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 03:53:29,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 03:53:29,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 03:53:29,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 03:53:29,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 03:53:29,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 03:53:29,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 03:53:29,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 03:53:29,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 03:53:29,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 03:53:29,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 03:53:29,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 03:53:29,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 03:53:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 03:53:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 03:53:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 03:53:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 03:53:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 03:53:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 03:53:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 03:53:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 03:53:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 03:53:29,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 03:53:29,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 03:53:29,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 03:53:29,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 03:53:29,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 03:53:29,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 03:53:29,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 03:53:29,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 03:53:29,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 03:53:29,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-19 03:53:29,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-19 03:53:29,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-19 03:53:29,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-19 03:53:29,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-19 03:53:29,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-19 03:53:29,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-19 03:53:29,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-19 03:53:29,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-19 03:53:29,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-19 03:53:29,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-19 03:53:29,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-19 03:53:29,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-19 03:53:29,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-19 03:53:29,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-19 03:53:29,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-19 03:53:29,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-19 03:53:29,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-19 03:53:29,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-19 03:53:29,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-19 03:53:29,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-19 03:53:29,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-19 03:53:29,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-19 03:53:29,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-19 03:53:29,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-19 03:53:29,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-19 03:53:29,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-19 03:53:29,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-19 03:53:29,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-19 03:53:29,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-19 03:53:29,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-19 03:53:29,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-19 03:53:29,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-19 03:53:29,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-19 03:53:29,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-19 03:53:29,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-19 03:53:29,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-19 03:53:29,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-19 03:53:29,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-19 03:53:29,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-19 03:53:29,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-19 03:53:29,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-19 03:53:29,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-19 03:53:29,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-19 03:53:29,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-19 03:53:29,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-19 03:53:29,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-19 03:53:29,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-19 03:53:29,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-19 03:53:29,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-19 03:53:29,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-19 03:53:29,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-19 03:53:29,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-19 03:53:29,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-19 03:53:29,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-19 03:53:29,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-19 03:53:29,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-19 03:53:29,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-19 03:53:29,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-19 03:53:29,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-19 03:53:29,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-19 03:53:29,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-19 03:53:29,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-19 03:53:29,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-19 03:53:29,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-19 03:53:29,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-19 03:53:29,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-19 03:53:29,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-19 03:53:29,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-19 03:53:29,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-19 03:53:29,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-19 03:53:29,708 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 03:53:29,708 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-19 03:53:29,708 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-19 03:53:30,300 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 881 ms to try 5000 different iterations. Found a solution that moves 4748 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.176966469725086. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03861261492925301, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05677450622134806, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5275555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03861261492925301, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17233333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23708333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 03:53:30,319 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2059126585:81 , srv1902894725:82 , srv679942789:82 , srv894909562:82 , srv1270964540:83 , srv1314468690:83 , srv1425799447:83 , srv1545853197:83 , srv1684459445:83 , srv1910230112:83 , srv465588354:83 , srv802415169:83 , srv9028390:83 , srv1180196123:84 , srv1222259493:84 , srv1300718705:84 , srv1388533565:84 , srv1771252934:84 , srv2026177989:84 , srv2070835034:84 , srv433015200:84 , srv624022252:84 , srv1065597400:85 , srv1125492022:85 , srv1157733758:85 , srv1225832885:85 , srv1344116240:85 , srv1473159306:85 , srv1903772771:85 , srv407592735:85 , srv427497669:85 , srv471542568:85 , srv575910455:85 , srv727377218:85 , srv804104983:85 , srv1000418383:86 , srv1070484404:86 , srv1173590366:86 , srv1273436778:86 , srv1277088888:86 , srv1282259358:86 , srv1291852958:86 , srv1362952210:86 , srv1435815688:86 , srv154892824:86 , srv1601576907:86 , srv1832548070:86 , srv1891843772:86 , srv1906940622:86 , srv198647643:86 , srv309079506:86 , srv331016813:86 , srv379082845:86 , srv486926314:86 , srv507307106:86 , srv859413696:86 , srv97720683:86 , srv1016625663:87 , srv1479791780:87 , srv154879625:87 , srv2101933639:87 , srv2123697544:87 , srv304488573:87 , srv588445009:87 , srv589492453:87 , srv609055584:87 , srv1506641423:88 , srv1707251103:88 , srv1969881595:88 , srv2109222463:88 , srv38481908:88 , srv399301600:88 , srv446868888:88 , srv6482941:88 , srv666604760:88 , srv789579626:88 , srv963418293:88 , srv1271917480:89 , srv1291738677:89 , srv1519150796:89 , srv1618064329:89 , srv1888906448:89 , srv2082661444:89 , srv2117462791:89 , srv812740927:89 , srv1066067229:90 , srv1211798173:90 , srv1365766994:90 , srv1439809976:90 , srv2097244291:90 , srv662770599:90 , srv72469254:90 , srv1132468577:91 , srv1310955516:91 , srv1341729207:91 , srv2012890043:92 , srv1678257363:93 , srv610608521:93 , srv1513039942:96 , srv450351636:433 } 2023-10-19 03:53:30,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 03:53:30,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1545853197=36, srv2109222463=61, srv1425799447=28, srv1969881595=52, srv894909562=96, srv1065597400=2, srv1271917480=14, srv963418293=98, srv2123697544=63, srv379082845=67, srv450351636=74, srv2097244291=59, srv331016813=66, srv1618064329=40, srv1273436778=15, srv1678257363=41, srv1707251103=43, srv1066067229=3, srv1435815688=29, srv407592735=70, srv1479791780=32, srv304488573=64, srv97720683=99, srv1157733758=7, srv507307106=78, srv1506641423=33, srv1362952210=25, srv446868888=73, srv38481908=68, srv2059126585=56, srv427497669=71, srv1903772771=49, srv859413696=95, srv198647643=53, srv1180196123=9, srv1270964540=13, srv1300718705=20, srv1291738677=18, srv789579626=91, srv2082661444=58, srv1225832885=12, srv1125492022=5, srv588445009=80, srv1016625663=1, srv1513039942=34, srv1365766994=26, srv486926314=77, srv1291852958=19, srv1310955516=21, srv727377218=90, srv433015200=72, srv1771252934=44, srv2117462791=62, srv1282259358=17, srv1891843772=47, srv1388533565=27, srv1684459445=42, srv2026177989=55, srv1519150796=35, srv2101933639=60, srv610608521=83, srv1173590366=8, srv309079506=65, srv624022252=84, srv72469254=89, srv154879625=37, srv465588354=75, srv399301600=69, srv1000418383=0, srv1132468577=6, srv812740927=94, srv1888906448=46, srv9028390=97, srv2012890043=54, srv1902894725=48, srv1473159306=31, srv154892824=38, srv1341729207=23, srv1222259493=11, srv1910230112=51, srv1832548070=45, srv804104983=93, srv1344116240=24, srv1070484404=4, srv802415169=92, srv2070835034=57, srv6482941=85, srv609055584=82, srv589492453=81, srv471542568=76, srv679942789=88, srv1906940622=50, srv1314468690=22, srv1601576907=39, srv1439809976=30, srv666604760=87, srv575910455=79, srv1277088888=16, srv662770599=86, srv1211798173=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 03:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 03:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 03:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 03:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 03:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 03:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 03:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 03:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 03:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 03:53:30,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 03:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 03:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 03:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 03:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 03:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 03:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 03:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 03:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 03:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 03:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 03:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 03:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 03:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 03:53:30,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 03:53:30,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 03:53:30,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 03:53:30,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 03:53:30,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 03:53:30,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 03:53:30,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 03:53:30,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-19 03:53:30,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-19 03:53:30,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-19 03:53:30,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-19 03:53:30,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-19 03:53:30,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-19 03:53:30,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-19 03:53:30,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-19 03:53:30,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-19 03:53:30,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-19 03:53:30,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-19 03:53:30,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-19 03:53:30,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-19 03:53:30,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-19 03:53:30,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-19 03:53:30,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-19 03:53:30,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-19 03:53:30,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-19 03:53:30,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-19 03:53:30,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-19 03:53:30,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-19 03:53:30,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-19 03:53:30,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-19 03:53:30,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-19 03:53:30,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-19 03:53:30,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-19 03:53:30,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-19 03:53:30,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-19 03:53:30,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-19 03:53:30,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-19 03:53:30,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-19 03:53:30,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-19 03:53:30,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-19 03:53:30,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-19 03:53:30,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-19 03:53:30,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-19 03:53:30,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-19 03:53:30,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-19 03:53:30,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-19 03:53:30,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-19 03:53:30,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-19 03:53:30,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-19 03:53:30,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-19 03:53:30,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-19 03:53:30,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-19 03:53:30,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-19 03:53:30,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-19 03:53:30,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-19 03:53:30,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-19 03:53:30,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-19 03:53:30,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-19 03:53:30,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-19 03:53:30,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-19 03:53:30,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-19 03:53:30,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-19 03:53:30,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-19 03:53:30,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-19 03:53:30,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-19 03:53:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-19 03:53:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-19 03:53:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-19 03:53:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-19 03:53:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-19 03:53:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-19 03:53:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-19 03:53:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-19 03:53:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-19 03:53:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-19 03:53:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-19 03:53:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-19 03:53:30,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-19 03:53:30,542 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03861261492925301, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05677450622134806, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03861261492925301, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17233333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23708333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 03:53:30,543 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-19 03:53:30,543 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17693322351850058, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03861261492925301, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05677450622134806, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03861261492925301, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17233333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23708333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-19 03:53:30,849 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 483 ms to try 5000 different iterations. Found a solution that moves 2618 regions; Going from a computed imbalance of 0.17693322351850058 to a new imbalance of 0.015409944139723117. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021599005117657097); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034530683829044846, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29088888888888886, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021599005117657097); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012166666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 03:53:30,863 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1513039942:74 , srv1678257363:79 , srv1618064329:80 , srv2012890043:80 , srv1132468577:81 , srv1211798173:81 , srv1291738677:81 , srv1341729207:81 , srv2097244291:81 , srv610608521:81 , srv1066067229:82 , srv1271917480:82 , srv1310955516:82 , srv1365766994:82 , srv1439809976:82 , srv1519150796:82 , srv399301600:82 , srv72469254:82 , srv1707251103:83 , srv2082661444:83 , srv2117462791:83 , srv38481908:83 , srv812740927:83 , srv1125492022:84 , srv1506641423:84 , srv154892824:84 , srv1888906448:84 , srv1891843772:84 , srv2109222463:84 , srv666604760:84 , srv1016625663:85 , srv1070484404:85 , srv1969881595:85 , srv198647643:85 , srv2101933639:85 , srv331016813:85 , srv446868888:85 , srv507307106:85 , srv588445009:85 , srv589492453:85 , srv6482941:85 , srv662770599:85 , srv963418293:85 , srv1000418383:86 , srv1157733758:86 , srv1225832885:86 , srv1282259358:86 , srv1291852958:86 , srv154879625:86 , srv1601576907:86 , srv1832548070:86 , srv1906940622:86 , srv2123697544:86 , srv407592735:86 , srv427497669:86 , srv486926314:86 , srv624022252:86 , srv789579626:86 , srv804104983:86 , srv1173590366:87 , srv1222259493:87 , srv1314468690:87 , srv1344116240:87 , srv1362952210:87 , srv1435815688:87 , srv1473159306:87 , srv1479791780:87 , srv1545853197:87 , srv1910230112:87 , srv2026177989:87 , srv2070835034:87 , srv304488573:87 , srv379082845:87 , srv471542568:87 , srv575910455:87 , srv609055584:87 , srv97720683:87 , srv1270964540:88 , srv1273436778:88 , srv1300718705:88 , srv1388533565:88 , srv1425799447:88 , srv1771252934:88 , srv309079506:88 , srv859413696:88 , srv1065597400:89 , srv1277088888:89 , srv1903772771:89 , srv465588354:89 , srv727377218:89 , srv1180196123:90 , srv1684459445:90 , srv1902894725:90 , srv2059126585:90 , srv679942789:90 , srv802415169:90 , srv9028390:90 , srv894909562:91 , srv433015200:92 , srv450351636:533 } 2023-10-19 03:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 03:53:30,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1545853197=36, srv2109222463=61, srv1425799447=28, srv1969881595=52, srv894909562=96, srv1065597400=2, srv1271917480=14, srv963418293=98, srv2123697544=63, srv379082845=67, srv450351636=74, srv2097244291=59, srv331016813=66, srv1618064329=40, srv1273436778=15, srv1678257363=41, srv1707251103=43, srv1066067229=3, srv1435815688=29, srv407592735=70, srv1479791780=32, srv304488573=64, srv97720683=99, srv1157733758=7, srv507307106=78, srv1506641423=33, srv1362952210=25, srv446868888=73, srv38481908=68, srv2059126585=56, srv427497669=71, srv1903772771=49, srv859413696=95, srv198647643=53, srv1180196123=9, srv1270964540=13, srv1300718705=20, srv1291738677=18, srv789579626=91, srv2082661444=58, srv1225832885=12, srv1125492022=5, srv588445009=80, srv1016625663=1, srv1513039942=34, srv1365766994=26, srv486926314=77, srv1291852958=19, srv1310955516=21, srv727377218=90, srv433015200=72, srv1771252934=44, srv2117462791=62, srv1282259358=17, srv1891843772=47, srv1388533565=27, srv1684459445=42, srv2026177989=55, srv1519150796=35, srv2101933639=60, srv610608521=83, srv1173590366=8, srv309079506=65, srv624022252=84, srv72469254=89, srv154879625=37, srv465588354=75, srv399301600=69, srv1000418383=0, srv1132468577=6, srv812740927=94, srv1888906448=46, srv9028390=97, srv2012890043=54, srv1902894725=48, srv1473159306=31, srv154892824=38, srv1341729207=23, srv1222259493=11, srv1910230112=51, srv1832548070=45, srv804104983=93, srv1344116240=24, srv1070484404=4, srv802415169=92, srv2070835034=57, srv6482941=85, srv609055584=82, srv589492453=81, srv471542568=76, srv679942789=88, srv1906940622=50, srv1314468690=22, srv1601576907=39, srv1439809976=30, srv666604760=87, srv575910455=79, srv1277088888=16, srv662770599=86, srv1211798173=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 03:53:30,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 03:53:30,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 03:53:30,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 03:53:30,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 03:53:30,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 03:53:30,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 03:53:30,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 03:53:30,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 03:53:30,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 03:53:30,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 03:53:30,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 03:53:30,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 03:53:30,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 03:53:30,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 03:53:30,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 03:53:30,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 03:53:30,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 03:53:30,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 03:53:30,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 03:53:30,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 03:53:30,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 03:53:30,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 03:53:30,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 03:53:30,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 03:53:30,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 03:53:30,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 03:53:30,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 03:53:30,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 03:53:30,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 03:53:30,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 03:53:30,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-19 03:53:30,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-19 03:53:30,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-19 03:53:30,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-19 03:53:30,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-19 03:53:30,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-19 03:53:30,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-19 03:53:30,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-19 03:53:30,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-19 03:53:30,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-19 03:53:30,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-19 03:53:30,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-19 03:53:30,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-19 03:53:30,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-19 03:53:30,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-19 03:53:30,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-19 03:53:30,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-19 03:53:30,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-19 03:53:30,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-19 03:53:30,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-19 03:53:30,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-19 03:53:30,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-19 03:53:30,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-19 03:53:30,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-19 03:53:30,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-19 03:53:30,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-19 03:53:30,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-19 03:53:30,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-19 03:53:30,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-19 03:53:30,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-19 03:53:30,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-19 03:53:30,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-19 03:53:30,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-19 03:53:30,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-19 03:53:30,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-19 03:53:30,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-19 03:53:30,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-19 03:53:30,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-19 03:53:30,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-19 03:53:30,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-19 03:53:30,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-19 03:53:30,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-19 03:53:30,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-19 03:53:30,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-19 03:53:30,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-19 03:53:30,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-19 03:53:30,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-19 03:53:30,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-19 03:53:30,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-19 03:53:30,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-19 03:53:30,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-19 03:53:30,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-19 03:53:30,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-19 03:53:30,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-19 03:53:30,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-19 03:53:30,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-19 03:53:30,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-19 03:53:30,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-19 03:53:30,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-19 03:53:30,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-19 03:53:30,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-19 03:53:30,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-19 03:53:30,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-19 03:53:30,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-19 03:53:30,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-19 03:53:30,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-19 03:53:30,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-19 03:53:30,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-19 03:53:30,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-19 03:53:30,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-19 03:53:30,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-19 03:53:31,070 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021599005117657097); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034530683829044846, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021599005117657097); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012166666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 03:53:31,071 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-19 03:53:31,071 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015391612511913379, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021599005117657097); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034530683829044846, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021599005117657097); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012166666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-19 03:53:31,358 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 436 ms to try 5000 different iterations. Found a solution that moves 988 regions; Going from a computed imbalance of 0.015391612511913379 to a new imbalance of 9.621745943620231E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006330861746253887); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007106690545187014); MoveCostFunction : (multiplier=7.0, imbalance=0.10977777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006330861746253887); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 03:53:31,359 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv433015200:82 , srv662770599:82 , srv859413696:82 , srv1273436778:83 , srv1277088888:83 , srv309079506:83 , srv727377218:83 , srv1065597400:84 , srv1180196123:84 , srv1435815688:84 , srv1684459445:84 , srv1888906448:84 , srv1969881595:84 , srv304488573:84 , srv588445009:84 , srv609055584:84 , srv6482941:84 , srv666604760:84 , srv789579626:84 , srv894909562:84 , srv963418293:84 , srv1016625663:85 , srv1066067229:85 , srv1132468577:85 , srv1173590366:85 , srv1282259358:85 , srv1291852958:85 , srv1300718705:85 , srv1310955516:85 , srv1341729207:85 , srv1362952210:85 , srv1365766994:85 , srv1388533565:85 , srv1439809976:85 , srv1479791780:85 , srv154879625:85 , srv1601576907:85 , srv1771252934:85 , srv1902894725:85 , srv1903772771:85 , srv1906940622:85 , srv2012890043:85 , srv2070835034:85 , srv2082661444:85 , srv2109222463:85 , srv2117462791:85 , srv2123697544:85 , srv379082845:85 , srv446868888:85 , srv575910455:85 , srv610608521:85 , srv72469254:85 , srv802415169:85 , srv812740927:85 , srv9028390:85 , srv97720683:85 , srv1000418383:86 , srv1070484404:86 , srv1211798173:86 , srv1222259493:86 , srv1271917480:86 , srv1291738677:86 , srv1344116240:86 , srv1425799447:86 , srv1473159306:86 , srv1506641423:86 , srv1678257363:86 , srv1832548070:86 , srv2059126585:86 , srv331016813:86 , srv38481908:86 , srv407592735:86 , srv427497669:86 , srv465588354:86 , srv471542568:86 , srv486926314:86 , srv589492453:86 , srv1157733758:87 , srv1225832885:87 , srv1270964540:87 , srv1314468690:87 , srv1513039942:87 , srv1519150796:87 , srv1618064329:87 , srv1707251103:87 , srv1891843772:87 , srv1910230112:87 , srv198647643:87 , srv2026177989:87 , srv2097244291:87 , srv2101933639:87 , srv399301600:87 , srv507307106:87 , srv624022252:87 , srv679942789:87 , srv804104983:87 , srv1125492022:88 , srv1545853197:88 , srv154892824:88 , srv450351636:548 } 2023-10-19 03:53:31,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 03:53:31,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1545853197=36, srv2109222463=61, srv1425799447=28, srv1969881595=52, srv894909562=96, srv1065597400=2, srv1271917480=14, srv963418293=98, srv2123697544=63, srv379082845=67, srv450351636=74, srv2097244291=59, srv331016813=66, srv1618064329=40, srv1273436778=15, srv1678257363=41, srv1707251103=43, srv1066067229=3, srv1435815688=29, srv407592735=70, srv1479791780=32, srv304488573=64, srv97720683=99, srv1157733758=7, srv507307106=78, srv1506641423=33, srv1362952210=25, srv446868888=73, srv38481908=68, srv2059126585=56, srv427497669=71, srv1903772771=49, srv859413696=95, srv198647643=53, srv1180196123=9, srv1270964540=13, srv1300718705=20, srv1291738677=18, srv789579626=91, srv2082661444=58, srv1225832885=12, srv1125492022=5, srv588445009=80, srv1016625663=1, srv1513039942=34, srv1365766994=26, srv486926314=77, srv1291852958=19, srv1310955516=21, srv727377218=90, srv433015200=72, srv1771252934=44, srv2117462791=62, srv1282259358=17, srv1891843772=47, srv1388533565=27, srv1684459445=42, srv2026177989=55, srv1519150796=35, srv2101933639=60, srv610608521=83, srv1173590366=8, srv309079506=65, srv624022252=84, srv72469254=89, srv154879625=37, srv465588354=75, srv399301600=69, srv1000418383=0, srv1132468577=6, srv812740927=94, srv1888906448=46, srv9028390=97, srv2012890043=54, srv1902894725=48, srv1473159306=31, srv154892824=38, srv1341729207=23, srv1222259493=11, srv1910230112=51, srv1832548070=45, srv804104983=93, srv1344116240=24, srv1070484404=4, srv802415169=92, srv2070835034=57, srv6482941=85, srv609055584=82, srv589492453=81, srv471542568=76, srv679942789=88, srv1906940622=50, srv1314468690=22, srv1601576907=39, srv1439809976=30, srv666604760=87, srv575910455=79, srv1277088888=16, srv662770599=86, srv1211798173=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 03:53:31,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 03:53:31,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 03:53:31,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 03:53:31,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 03:53:31,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 03:53:31,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 03:53:31,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 03:53:31,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 03:53:31,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 03:53:31,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 03:53:31,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 03:53:31,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 03:53:31,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 03:53:31,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 03:53:31,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 03:53:31,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 03:53:31,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 03:53:31,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 03:53:31,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 03:53:31,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 03:53:31,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 03:53:31,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 03:53:31,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 03:53:31,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 03:53:31,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 03:53:31,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 03:53:31,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 03:53:31,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 03:53:31,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 03:53:31,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 03:53:31,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-19 03:53:31,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-19 03:53:31,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-19 03:53:31,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-19 03:53:31,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-19 03:53:31,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-19 03:53:31,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-19 03:53:31,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-19 03:53:31,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-19 03:53:31,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-19 03:53:31,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-19 03:53:31,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-19 03:53:31,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-19 03:53:31,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-19 03:53:31,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-19 03:53:31,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-19 03:53:31,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-19 03:53:31,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-19 03:53:31,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-19 03:53:31,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-19 03:53:31,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-19 03:53:31,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-19 03:53:31,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-19 03:53:31,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-19 03:53:31,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-19 03:53:31,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-19 03:53:31,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-19 03:53:31,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-19 03:53:31,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-19 03:53:31,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-19 03:53:31,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-19 03:53:31,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-19 03:53:31,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-19 03:53:31,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-19 03:53:31,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-19 03:53:31,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-19 03:53:31,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-19 03:53:31,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-19 03:53:31,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-19 03:53:31,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-19 03:53:31,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-19 03:53:31,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-19 03:53:31,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-19 03:53:31,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-19 03:53:31,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-19 03:53:31,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-19 03:53:31,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-19 03:53:31,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-19 03:53:31,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-19 03:53:31,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-19 03:53:31,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-19 03:53:31,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-19 03:53:31,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-19 03:53:31,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-19 03:53:31,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-19 03:53:31,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-19 03:53:31,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-19 03:53:31,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-19 03:53:31,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-19 03:53:31,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-19 03:53:31,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-19 03:53:31,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-19 03:53:31,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-19 03:53:31,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-19 03:53:31,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-19 03:53:31,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-19 03:53:31,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-19 03:53:31,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-19 03:53:31,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-19 03:53:31,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-19 03:53:31,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-19 03:53:31,497 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006330861746253887); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007106690545187014); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006330861746253887); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 03:53:31,497 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-19 03:53:31,498 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.552564704979968E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006330861746253887); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007106690545187014); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006330861746253887); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-19 03:53:31,852 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 458 ms to try 5000 different iterations. Found a solution that moves 266 regions; Going from a computed imbalance of 9.552564704979968E-4 to a new imbalance of 5.019986743708681E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.909066927536782E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.029555555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.909066927536782E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 03:53:31,853 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1157733758:83 , srv1479791780:83 , srv2101933639:83 , srv465588354:83 , srv507307106:83 , srv610608521:83 , srv679942789:83 , srv1000418383:84 , srv1065597400:84 , srv1173590366:84 , srv1180196123:84 , srv1211798173:84 , srv1225832885:84 , srv1271917480:84 , srv1277088888:84 , srv1310955516:84 , srv1314468690:84 , srv1344116240:84 , srv1362952210:84 , srv1425799447:84 , srv1473159306:84 , srv1519150796:84 , srv1545853197:84 , srv154879625:84 , srv154892824:84 , srv1707251103:84 , srv1832548070:84 , srv1902894725:84 , srv1903772771:84 , srv2026177989:84 , srv2097244291:84 , srv2117462791:84 , srv2123697544:84 , srv304488573:84 , srv379082845:84 , srv407592735:84 , srv433015200:84 , srv471542568:84 , srv486926314:84 , srv589492453:84 , srv609055584:84 , srv6482941:84 , srv662770599:84 , srv789579626:84 , srv802415169:84 , srv812740927:84 , srv963418293:84 , srv97720683:84 , srv1016625663:85 , srv1066067229:85 , srv1070484404:85 , srv1125492022:85 , srv1132468577:85 , srv1222259493:85 , srv1270964540:85 , srv1291852958:85 , srv1341729207:85 , srv1365766994:85 , srv1388533565:85 , srv1435815688:85 , srv1439809976:85 , srv1506641423:85 , srv1513039942:85 , srv1601576907:85 , srv1678257363:85 , srv1684459445:85 , srv1771252934:85 , srv1888906448:85 , srv1891843772:85 , srv1906940622:85 , srv1969881595:85 , srv198647643:85 , srv2012890043:85 , srv2059126585:85 , srv2082661444:85 , srv2109222463:85 , srv309079506:85 , srv331016813:85 , srv38481908:85 , srv399301600:85 , srv427497669:85 , srv446868888:85 , srv575910455:85 , srv624022252:85 , srv727377218:85 , srv804104983:85 , srv894909562:85 , srv9028390:85 , srv1273436778:86 , srv1282259358:86 , srv1291738677:86 , srv1300718705:86 , srv1618064329:86 , srv1910230112:86 , srv2070835034:86 , srv588445009:86 , srv666604760:86 , srv72469254:86 , srv859413696:87 , srv450351636:628 } 2023-10-19 03:53:31,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 03:53:31,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1545853197=36, srv2109222463=61, srv1425799447=28, srv1969881595=52, srv894909562=96, srv1065597400=2, srv1271917480=14, srv963418293=98, srv2123697544=63, srv379082845=67, srv450351636=74, srv2097244291=59, srv331016813=66, srv1618064329=40, srv1273436778=15, srv1678257363=41, srv1707251103=43, srv1066067229=3, srv1435815688=29, srv407592735=70, srv1479791780=32, srv304488573=64, srv97720683=99, srv1157733758=7, srv507307106=78, srv1506641423=33, srv1362952210=25, srv446868888=73, srv38481908=68, srv2059126585=56, srv427497669=71, srv1903772771=49, srv859413696=95, srv198647643=53, srv1180196123=9, srv1270964540=13, srv1300718705=20, srv1291738677=18, srv789579626=91, srv2082661444=58, srv1225832885=12, srv1125492022=5, srv588445009=80, srv1016625663=1, srv1513039942=34, srv1365766994=26, srv486926314=77, srv1291852958=19, srv1310955516=21, srv727377218=90, srv433015200=72, srv1771252934=44, srv2117462791=62, srv1282259358=17, srv1891843772=47, srv1388533565=27, srv1684459445=42, srv2026177989=55, srv1519150796=35, srv2101933639=60, srv610608521=83, srv1173590366=8, srv309079506=65, srv624022252=84, srv72469254=89, srv154879625=37, srv465588354=75, srv399301600=69, srv1000418383=0, srv1132468577=6, srv812740927=94, srv1888906448=46, srv9028390=97, srv2012890043=54, srv1902894725=48, srv1473159306=31, srv154892824=38, srv1341729207=23, srv1222259493=11, srv1910230112=51, srv1832548070=45, srv804104983=93, srv1344116240=24, srv1070484404=4, srv802415169=92, srv2070835034=57, srv6482941=85, srv609055584=82, srv589492453=81, srv471542568=76, srv679942789=88, srv1906940622=50, srv1314468690=22, srv1601576907=39, srv1439809976=30, srv666604760=87, srv575910455=79, srv1277088888=16, srv662770599=86, srv1211798173=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 03:53:31,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 03:53:31,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 03:53:31,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 03:53:31,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 03:53:31,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 03:53:31,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 03:53:31,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 03:53:31,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 03:53:31,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 03:53:31,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 03:53:31,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 03:53:31,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 03:53:31,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 03:53:31,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 03:53:31,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 03:53:31,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 03:53:31,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 03:53:31,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 03:53:31,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 03:53:31,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 03:53:31,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 03:53:31,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 03:53:31,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 03:53:31,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 03:53:31,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 03:53:31,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 03:53:31,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 03:53:31,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 03:53:31,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 03:53:31,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 03:53:31,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-19 03:53:31,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-19 03:53:31,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-19 03:53:31,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-19 03:53:31,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-19 03:53:31,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-19 03:53:31,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-19 03:53:31,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-19 03:53:31,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-19 03:53:31,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-19 03:53:31,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-19 03:53:31,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-19 03:53:31,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-19 03:53:31,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-19 03:53:31,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-19 03:53:31,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-19 03:53:31,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-19 03:53:31,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-19 03:53:31,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-19 03:53:31,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-19 03:53:31,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-19 03:53:31,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-19 03:53:31,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-19 03:53:31,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-19 03:53:31,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-19 03:53:31,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-19 03:53:31,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-19 03:53:31,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-19 03:53:31,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-19 03:53:31,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-19 03:53:31,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-19 03:53:31,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-19 03:53:31,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-19 03:53:31,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-19 03:53:31,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-19 03:53:31,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-19 03:53:31,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-19 03:53:31,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-19 03:53:31,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-19 03:53:31,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-19 03:53:31,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-19 03:53:31,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-19 03:53:31,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-19 03:53:31,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-19 03:53:31,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-19 03:53:31,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-19 03:53:31,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-19 03:53:31,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-19 03:53:31,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-19 03:53:31,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-19 03:53:31,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-19 03:53:31,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-19 03:53:31,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-19 03:53:31,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-19 03:53:31,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-19 03:53:31,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-19 03:53:31,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-19 03:53:31,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-19 03:53:31,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-19 03:53:31,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-19 03:53:31,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-19 03:53:31,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-19 03:53:31,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-19 03:53:31,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-19 03:53:31,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-19 03:53:31,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-19 03:53:31,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-19 03:53:31,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-19 03:53:31,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-19 03:53:31,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-19 03:53:31,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-19 03:53:31,988 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.909066927536782E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.909066927536782E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 03:53:31,989 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-19 03:53:31,989 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.833729562754128E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.909066927536782E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.909066927536782E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-19 03:53:32,236 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 352 ms to try 5000 different iterations. Found a solution that moves 28 regions; Going from a computed imbalance of 4.833729562754128E-5 to a new imbalance of 7.583817690791035E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.003111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 03:53:32,237 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1270964540:83 , srv1273436778:83 , srv1282259358:83 , srv1300718705:83 , srv1506641423:83 , srv1678257363:83 , srv1903772771:83 , srv1910230112:83 , srv198647643:83 , srv446868888:83 , srv72469254:83 , srv804104983:83 , srv859413696:83 , srv9028390:83 , srv1000418383:84 , srv1016625663:84 , srv1065597400:84 , srv1066067229:84 , srv1070484404:84 , srv1125492022:84 , srv1132468577:84 , srv1173590366:84 , srv1180196123:84 , srv1222259493:84 , srv1225832885:84 , srv1277088888:84 , srv1291738677:84 , srv1291852958:84 , srv1310955516:84 , srv1341729207:84 , srv1344116240:84 , srv1362952210:84 , srv1365766994:84 , srv1388533565:84 , srv1435815688:84 , srv1439809976:84 , srv1473159306:84 , srv1479791780:84 , srv1513039942:84 , srv1519150796:84 , srv1545853197:84 , srv154879625:84 , srv154892824:84 , srv1601576907:84 , srv1618064329:84 , srv1684459445:84 , srv1707251103:84 , srv1771252934:84 , srv1832548070:84 , srv1888906448:84 , srv1891843772:84 , srv1906940622:84 , srv1969881595:84 , srv2012890043:84 , srv2026177989:84 , srv2059126585:84 , srv2070835034:84 , srv2082661444:84 , srv2097244291:84 , srv2101933639:84 , srv2109222463:84 , srv2123697544:84 , srv304488573:84 , srv309079506:84 , srv331016813:84 , srv379082845:84 , srv38481908:84 , srv399301600:84 , srv427497669:84 , srv433015200:84 , srv471542568:84 , srv486926314:84 , srv575910455:84 , srv588445009:84 , srv589492453:84 , srv609055584:84 , srv610608521:84 , srv624022252:84 , srv666604760:84 , srv679942789:84 , srv727377218:84 , srv789579626:84 , srv802415169:84 , srv894909562:84 , srv97720683:84 , srv1157733758:85 , srv1211798173:85 , srv1271917480:85 , srv1314468690:85 , srv1425799447:85 , srv1902894725:85 , srv2117462791:85 , srv407592735:85 , srv465588354:85 , srv507307106:85 , srv6482941:85 , srv662770599:85 , srv812740927:85 , srv963418293:85 , srv450351636:684 } 2023-10-19 03:53:32,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 03:53:32,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1545853197=36, srv2109222463=61, srv1425799447=28, srv1969881595=52, srv894909562=96, srv1065597400=2, srv1271917480=14, srv963418293=98, srv2123697544=63, srv379082845=67, srv450351636=74, srv2097244291=59, srv331016813=66, srv1618064329=40, srv1273436778=15, srv1678257363=41, srv1707251103=43, srv1066067229=3, srv1435815688=29, srv407592735=70, srv1479791780=32, srv304488573=64, srv97720683=99, srv1157733758=7, srv507307106=78, srv1506641423=33, srv1362952210=25, srv446868888=73, srv38481908=68, srv2059126585=56, srv427497669=71, srv1903772771=49, srv859413696=95, srv198647643=53, srv1180196123=9, srv1270964540=13, srv1300718705=20, srv1291738677=18, srv789579626=91, srv2082661444=58, srv1225832885=12, srv1125492022=5, srv588445009=80, srv1016625663=1, srv1513039942=34, srv1365766994=26, srv486926314=77, srv1291852958=19, srv1310955516=21, srv727377218=90, srv433015200=72, srv1771252934=44, srv2117462791=62, srv1282259358=17, srv1891843772=47, srv1388533565=27, srv1684459445=42, srv2026177989=55, srv1519150796=35, srv2101933639=60, srv610608521=83, srv1173590366=8, srv309079506=65, srv624022252=84, srv72469254=89, srv154879625=37, srv465588354=75, srv399301600=69, srv1000418383=0, srv1132468577=6, srv812740927=94, srv1888906448=46, srv9028390=97, srv2012890043=54, srv1902894725=48, srv1473159306=31, srv154892824=38, srv1341729207=23, srv1222259493=11, srv1910230112=51, srv1832548070=45, srv804104983=93, srv1344116240=24, srv1070484404=4, srv802415169=92, srv2070835034=57, srv6482941=85, srv609055584=82, srv589492453=81, srv471542568=76, srv679942789=88, srv1906940622=50, srv1314468690=22, srv1601576907=39, srv1439809976=30, srv666604760=87, srv575910455=79, srv1277088888=16, srv662770599=86, srv1211798173=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 03:53:32,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 03:53:32,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 03:53:32,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 03:53:32,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 03:53:32,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 03:53:32,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 03:53:32,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 03:53:32,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 03:53:32,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 03:53:32,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 03:53:32,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 03:53:32,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 03:53:32,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 03:53:32,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 03:53:32,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 03:53:32,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 03:53:32,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 03:53:32,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 03:53:32,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 03:53:32,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 03:53:32,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 03:53:32,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 03:53:32,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 03:53:32,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 03:53:32,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 03:53:32,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 03:53:32,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 03:53:32,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 03:53:32,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 03:53:32,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 03:53:32,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-19 03:53:32,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-19 03:53:32,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-19 03:53:32,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-19 03:53:32,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-19 03:53:32,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-19 03:53:32,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-19 03:53:32,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-19 03:53:32,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-19 03:53:32,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-19 03:53:32,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-19 03:53:32,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-19 03:53:32,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-19 03:53:32,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-19 03:53:32,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-19 03:53:32,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-19 03:53:32,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-19 03:53:32,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-19 03:53:32,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-19 03:53:32,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-19 03:53:32,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-19 03:53:32,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-19 03:53:32,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-19 03:53:32,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-19 03:53:32,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-19 03:53:32,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-19 03:53:32,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-19 03:53:32,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-19 03:53:32,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-19 03:53:32,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-19 03:53:32,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-19 03:53:32,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-19 03:53:32,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-19 03:53:32,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-19 03:53:32,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-19 03:53:32,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-19 03:53:32,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-19 03:53:32,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-19 03:53:32,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-19 03:53:32,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-19 03:53:32,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-19 03:53:32,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-19 03:53:32,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-19 03:53:32,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-19 03:53:32,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-19 03:53:32,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-19 03:53:32,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-19 03:53:32,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-19 03:53:32,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-19 03:53:32,378 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 03:53:32,379 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1270964540:83 , srv1273436778:83 , srv1282259358:83 , srv1300718705:83 , srv1506641423:83 , srv1678257363:83 , srv1903772771:83 , srv1910230112:83 , srv198647643:83 , srv446868888:83 , srv72469254:83 , srv804104983:83 , srv859413696:83 , srv9028390:83 , srv1000418383:84 , srv1016625663:84 , srv1065597400:84 , srv1066067229:84 , srv1070484404:84 , srv1125492022:84 , srv1132468577:84 , srv1173590366:84 , srv1180196123:84 , srv1222259493:84 , srv1225832885:84 , srv1277088888:84 , srv1291738677:84 , srv1291852958:84 , srv1310955516:84 , srv1341729207:84 , srv1344116240:84 , srv1362952210:84 , srv1365766994:84 , srv1388533565:84 , srv1435815688:84 , srv1439809976:84 , srv1473159306:84 , srv1479791780:84 , srv1513039942:84 , srv1519150796:84 , srv1545853197:84 , srv154879625:84 , srv154892824:84 , srv1601576907:84 , srv1618064329:84 , srv1684459445:84 , srv1707251103:84 , srv1771252934:84 , srv1832548070:84 , srv1888906448:84 , srv1891843772:84 , srv1906940622:84 , srv1969881595:84 , srv2012890043:84 , srv2026177989:84 , srv2059126585:84 , srv2070835034:84 , srv2082661444:84 , srv2097244291:84 , srv2101933639:84 , srv2109222463:84 , srv2123697544:84 , srv304488573:84 , srv309079506:84 , srv331016813:84 , srv379082845:84 , srv38481908:84 , srv399301600:84 , srv427497669:84 , srv433015200:84 , srv471542568:84 , srv486926314:84 , srv575910455:84 , srv588445009:84 , srv589492453:84 , srv609055584:84 , srv610608521:84 , srv624022252:84 , srv666604760:84 , srv679942789:84 , srv727377218:84 , srv789579626:84 , srv802415169:84 , srv894909562:84 , srv97720683:84 , srv1157733758:85 , srv1211798173:85 , srv1271917480:85 , srv1314468690:85 , srv1425799447:85 , srv1902894725:85 , srv2117462791:85 , srv407592735:85 , srv465588354:85 , srv507307106:85 , srv6482941:85 , srv662770599:85 , srv812740927:85 , srv963418293:85 , srv450351636:684 } 2023-10-19 03:53:32,406 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=327 (was 327), ProcessCount=169 (was 169), AvailableMemoryMB=10842 (was 11356) 2023-10-19 03:53:32,417 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=327, ProcessCount=169, AvailableMemoryMB=10842 2023-10-19 03:53:32,418 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-19 03:53:32,418 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-10-19 03:53:32,431 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv107810914:84 , srv1105857905:84 , srv1115690825:84 , srv124195059:84 , srv1346519152:84 , srv1399390641:84 , srv143743631:84 , srv1647800530:84 , srv1671551468:84 , srv1906456534:84 , srv1917831803:84 , srv1941062997:84 , srv199922615:84 , srv2017828278:84 , srv2047203058:84 , srv2105191769:84 , srv424672714:84 , srv60928060:84 , srv622149559:84 , srv649122442:84 , srv653378403:84 , srv665410974:84 , srv702001141:84 , srv822668083:84 , srv841810532:84 , srv863693790:84 , srv89221065:84 , srv944902565:84 , srv991825183:84 , srv626396818:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-10-19 03:53:32,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 03:53:32,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1105857905=1, srv2105191769=15, srv944902565=28, srv626396818=19, srv991825183=29, srv1917831803=10, srv107810914=0, srv1399390641=5, srv665410974=22, srv424672714=16, srv199922615=12, srv649122442=20, srv1115690825=2, srv841810532=25, srv1647800530=7, srv2047203058=14, srv863693790=26, srv1906456534=9, srv1346519152=4, srv653378403=21, srv124195059=3, srv1941062997=11, srv143743631=6, srv822668083=24, srv622149559=18, srv2017828278=13, srv60928060=17, srv702001141=23, srv89221065=27, srv1671551468=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 03:53:32,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 03:53:32,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 03:53:32,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 03:53:32,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 03:53:32,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 03:53:32,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 03:53:32,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 03:53:32,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 03:53:32,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 03:53:32,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 03:53:32,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 03:53:32,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 03:53:32,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 03:53:32,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 03:53:32,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 03:53:32,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 03:53:32,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 03:53:32,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 03:53:32,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 03:53:32,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 03:53:32,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 03:53:32,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 03:53:32,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 03:53:32,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 03:53:32,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 03:53:32,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 03:53:32,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 03:53:32,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 03:53:32,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 03:53:32,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 03:53:32,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-19 03:53:32,458 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.111167740858637, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 03:53:32,459 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912543329185765, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.111167740858637, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-10-19 03:55:32,446 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5665607 different iterations. Found a solution that moves 1855 regions; Going from a computed imbalance of 0.9912543329185765 to a new imbalance of 4.329662539733032E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.687037037037037, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 03:55:32,455 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv107810914:90 , srv1105857905:90 , srv1115690825:90 , srv124195059:90 , srv1346519152:90 , srv1399390641:90 , srv143743631:90 , srv1647800530:90 , srv1671551468:90 , srv1906456534:90 , srv1917831803:90 , srv1941062997:90 , srv199922615:90 , srv2017828278:90 , srv2047203058:90 , srv2105191769:90 , srv424672714:90 , srv60928060:90 , srv622149559:90 , srv626396818:90 , srv649122442:90 , srv653378403:90 , srv665410974:90 , srv702001141:90 , srv822668083:90 , srv841810532:90 , srv863693790:90 , srv89221065:90 , srv944902565:90 , srv991825183:90 } 2023-10-19 03:55:32,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 03:55:32,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1105857905=1, srv2105191769=15, srv944902565=28, srv626396818=19, srv991825183=29, srv1917831803=10, srv107810914=0, srv1399390641=5, srv665410974=22, srv424672714=16, srv199922615=12, srv649122442=20, srv1115690825=2, srv841810532=25, srv1647800530=7, srv2047203058=14, srv863693790=26, srv1906456534=9, srv1346519152=4, srv653378403=21, srv124195059=3, srv1941062997=11, srv143743631=6, srv822668083=24, srv622149559=18, srv2017828278=13, srv60928060=17, srv702001141=23, srv89221065=27, srv1671551468=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 03:55:32,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 03:55:32,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 03:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 03:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 03:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 03:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 03:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 03:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 03:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 03:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 03:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 03:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 03:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 03:55:32,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 03:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 03:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 03:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 03:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 03:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 03:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 03:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 03:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 03:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 03:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 03:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 03:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 03:55:32,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 03:55:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 03:55:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 03:55:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 03:55:32,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-19 03:55:32,482 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 03:55:32,482 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv107810914:90 , srv1105857905:90 , srv1115690825:90 , srv124195059:90 , srv1346519152:90 , srv1399390641:90 , srv143743631:90 , srv1647800530:90 , srv1671551468:90 , srv1906456534:90 , srv1917831803:90 , srv1941062997:90 , srv199922615:90 , srv2017828278:90 , srv2047203058:90 , srv2105191769:90 , srv424672714:90 , srv60928060:90 , srv622149559:90 , srv626396818:90 , srv649122442:90 , srv653378403:90 , srv665410974:90 , srv702001141:90 , srv822668083:90 , srv841810532:90 , srv863693790:90 , srv89221065:90 , srv944902565:90 , srv991825183:90 } 2023-10-19 03:55:32,495 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=206 (was 327), ProcessCount=167 (was 169), AvailableMemoryMB=10601 (was 10842)