2023-10-20 22:53:27,850 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-10-20 22:53:28,160 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-20 22:53:28,264 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-10-20 22:53:28,332 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-20 22:53:28,342 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-20 22:53:28,357 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=357, ProcessCount=169, AvailableMemoryMB=9185 2023-10-20 22:53:28,358 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-20 22:53:28,359 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-20 22:53:28,496 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1003992243:84 , srv1049013604:84 , srv1067232347:84 , srv1081629712:84 , srv1106969105:84 , srv1158771191:84 , srv115955428:84 , srv1231189006:84 , srv1266703440:84 , srv1293399079:84 , srv1317181188:84 , srv1346380814:84 , srv1383120285:84 , srv1396438687:84 , srv1402276609:84 , srv140516759:84 , srv1405271699:84 , srv1407486881:84 , srv1442181191:84 , srv1443647901:84 , srv1455855722:84 , srv1481069114:84 , srv1483642240:84 , srv1488261827:84 , srv1533106987:84 , srv1562717304:84 , srv1573530059:84 , srv1624934652:84 , srv1685157820:84 , srv1705150531:84 , srv1711097026:84 , srv171778577:84 , srv1764695934:84 , srv1794757379:84 , srv1852882665:84 , srv1858142486:84 , srv1865654517:84 , srv1866694642:84 , srv1882416371:84 , srv1961258881:84 , srv2005280046:84 , srv2018496474:84 , srv203038009:84 , srv2038360206:84 , srv2054429522:84 , srv2067757456:84 , srv2069260113:84 , srv2070005649:84 , srv2090088428:84 , srv215949714:84 , srv240830518:84 , srv261663032:84 , srv262537214:84 , srv3538548:84 , srv371000685:84 , srv390581881:84 , srv398817152:84 , srv418547670:84 , srv418569539:84 , srv429628147:84 , srv452417545:84 , srv453415199:84 , srv463575785:84 , srv466818714:84 , srv467479739:84 , srv480282557:84 , srv499012744:84 , srv503016794:84 , srv550975866:84 , srv584025241:84 , srv600724505:84 , srv611578354:84 , srv619051312:84 , srv622249421:84 , srv622594928:84 , srv645833509:84 , srv646881806:84 , srv70431370:84 , srv706995183:84 , srv730765455:84 , srv750352678:84 , srv764490360:84 , srv791831348:84 , srv792262031:84 , srv799656349:84 , srv809457258:84 , srv809702192:84 , srv817709116:84 , srv860948605:84 , srv8987161:84 , srv910626609:84 , srv91472899:84 , srv919504532:84 , srv947917814:84 , srv957372773:84 , srv963026301:84 , srv967891314:84 , srv97514374:84 , srv977064743:84 , srv306981408:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-10-20 22:53:28,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-20 22:53:28,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1961258881=39, srv622594928=75, srv791831348=83, srv1882416371=38, srv750352678=81, srv8987161=90, srv764490360=82, srv1407486881=17, srv611578354=72, srv622249421=74, srv1488261827=23, srv967891314=97, srv467479739=65, srv97514374=98, srv1624934652=27, srv262537214=52, srv1483642240=22, srv706995183=79, srv1003992243=0, srv860948605=89, srv2054429522=44, srv1764695934=32, srv1865654517=36, srv1866694642=37, srv1106969105=4, srv499012744=67, srv1794757379=33, srv947917814=94, srv799656349=85, srv919504532=93, srv963026301=96, srv215949714=49, srv1573530059=26, srv1705150531=29, srv452417545=61, srv418547670=58, srv1396438687=13, srv2038360206=43, srv2090088428=48, srv1562717304=25, srv466818714=64, srv792262031=84, srv2005280046=40, srv1481069114=21, srv261663032=51, srv646881806=77, srv390581881=56, srv1266703440=8, srv957372773=95, srv2018496474=41, srv3538548=54, srv1852882665=34, srv1293399079=9, srv910626609=91, srv584025241=70, srv463575785=63, srv503016794=68, srv645833509=76, srv1383120285=12, srv1081629712=3, srv1402276609=14, srv1711097026=30, srv398817152=57, srv809457258=86, srv453415199=62, srv70431370=78, srv203038009=42, srv1317181188=10, srv2069260113=46, srv1158771191=5, srv1231189006=7, srv1443647901=19, srv2070005649=47, srv429628147=60, srv619051312=73, srv1049013604=1, srv730765455=80, srv1405271699=16, srv600724505=71, srv240830518=50, srv418569539=59, srv977064743=99, srv371000685=55, srv1858142486=35, srv480282557=66, srv809702192=87, srv115955428=6, srv1442181191=18, srv1067232347=2, srv140516759=15, srv1455855722=20, srv1533106987=24, srv550975866=69, srv1685157820=28, srv91472899=92, srv2067757456=45, srv306981408=53, srv171778577=31, srv1346380814=11, srv817709116=88} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-20 22:53:28,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-20 22:53:28,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-20 22:53:28,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-20 22:53:28,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-20 22:53:28,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-20 22:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-20 22:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-20 22:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-20 22:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-20 22:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-20 22:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-20 22:53:28,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-20 22:53:28,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-20 22:53:28,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-20 22:53:28,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-20 22:53:28,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-20 22:53:28,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-20 22:53:28,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-20 22:53:28,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-20 22:53:28,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-20 22:53:28,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-20 22:53:28,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-20 22:53:28,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-20 22:53:28,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-20 22:53:28,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-20 22:53:28,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-20 22:53:28,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-20 22:53:28,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-20 22:53:28,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-20 22:53:28,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-20 22:53:28,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-20 22:53:28,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-20 22:53:28,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-20 22:53:28,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-20 22:53:28,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-20 22:53:28,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-20 22:53:28,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-20 22:53:28,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-20 22:53:28,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-20 22:53:28,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-20 22:53:28,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-20 22:53:28,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-20 22:53:28,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-20 22:53:28,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-20 22:53:28,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-20 22:53:28,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-20 22:53:28,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-20 22:53:28,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-20 22:53:28,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-20 22:53:28,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-20 22:53:28,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-20 22:53:28,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-20 22:53:28,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-20 22:53:28,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-20 22:53:28,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-20 22:53:28,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-20 22:53:28,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-20 22:53:28,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-20 22:53:28,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-20 22:53:28,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-20 22:53:28,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-20 22:53:28,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-20 22:53:28,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-20 22:53:28,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-20 22:53:28,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-20 22:53:28,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-20 22:53:28,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-20 22:53:28,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-20 22:53:28,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-20 22:53:28,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-20 22:53:28,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-20 22:53:28,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-20 22:53:28,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-20 22:53:28,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-20 22:53:28,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-20 22:53:28,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-20 22:53:28,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-20 22:53:28,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-20 22:53:28,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-20 22:53:28,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-20 22:53:28,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-20 22:53:28,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-20 22:53:28,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-20 22:53:28,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-20 22:53:28,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-20 22:53:28,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-20 22:53:28,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-20 22:53:28,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-20 22:53:28,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-20 22:53:28,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-20 22:53:28,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-20 22:53:28,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-20 22:53:28,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-20 22:53:28,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-20 22:53:28,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-20 22:53:28,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-20 22:53:28,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-20 22:53:28,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-20 22:53:28,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-20 22:53:28,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-20 22:53:28,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-20 22:53:29,002 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 22:53:29,002 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-20 22:53:29,002 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-20 22:53:29,492 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 827 ms to try 5000 different iterations. Found a solution that moves 4653 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18389058160961863. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03993510136131179, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.052059708446615185, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.517, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03993510136131179, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17941666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24333333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 22:53:29,511 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2054429522:80 , srv1705150531:81 , srv1573530059:82 , srv1624934652:82 , srv1764695934:82 , srv1442181191:83 , srv957372773:83 , srv1081629712:84 , srv1266703440:84 , srv1317181188:84 , srv1402276609:84 , srv1685157820:84 , srv1711097026:84 , srv171778577:84 , srv1961258881:84 , srv467479739:84 , srv809702192:84 , srv947917814:84 , srv1067232347:85 , srv1383120285:85 , srv1396438687:85 , srv140516759:85 , srv1407486881:85 , srv1443647901:85 , srv1483642240:85 , srv1562717304:85 , srv1882416371:85 , srv2090088428:85 , srv390581881:85 , srv398817152:85 , srv429628147:85 , srv452417545:85 , srv463575785:85 , srv480282557:85 , srv600724505:85 , srv611578354:85 , srv645833509:85 , srv750352678:85 , srv792262031:85 , srv910626609:85 , srv963026301:85 , srv1488261827:86 , srv1865654517:86 , srv2005280046:86 , srv2018496474:86 , srv2038360206:86 , srv261663032:86 , srv262537214:86 , srv418569539:86 , srv453415199:86 , srv817709116:86 , srv97514374:86 , srv1003992243:87 , srv1158771191:87 , srv1293399079:87 , srv1405271699:87 , srv1533106987:87 , srv1858142486:87 , srv2067757456:87 , srv2069260113:87 , srv466818714:87 , srv622249421:87 , srv70431370:87 , srv706995183:87 , srv791831348:87 , srv809457258:87 , srv8987161:87 , srv977064743:87 , srv115955428:88 , srv1231189006:88 , srv1455855722:88 , srv1481069114:88 , srv1866694642:88 , srv203038009:88 , srv2070005649:88 , srv215949714:88 , srv371000685:88 , srv499012744:88 , srv503016794:88 , srv584025241:88 , srv730765455:88 , srv799656349:88 , srv91472899:88 , srv967891314:88 , srv1049013604:89 , srv1346380814:89 , srv1794757379:89 , srv240830518:89 , srv619051312:89 , srv646881806:89 , srv1852882665:90 , srv418547670:90 , srv1106969105:91 , srv3538548:91 , srv550975866:91 , srv919504532:91 , srv622594928:93 , srv764490360:93 , srv860948605:93 , srv306981408:445 } 2023-10-20 22:53:29,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-20 22:53:29,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1961258881=39, srv622594928=75, srv791831348=83, srv1882416371=38, srv750352678=81, srv8987161=90, srv764490360=82, srv1407486881=17, srv611578354=72, srv622249421=74, srv1488261827=23, srv967891314=97, srv467479739=65, srv97514374=98, srv1624934652=27, srv262537214=52, srv1483642240=22, srv706995183=79, srv1003992243=0, srv860948605=89, srv2054429522=44, srv1764695934=32, srv1865654517=36, srv1866694642=37, srv1106969105=4, srv499012744=67, srv1794757379=33, srv947917814=94, srv799656349=85, srv919504532=93, srv963026301=96, srv215949714=49, srv1573530059=26, srv1705150531=29, srv452417545=61, srv418547670=58, srv1396438687=13, srv2038360206=43, srv2090088428=48, srv1562717304=25, srv466818714=64, srv792262031=84, srv2005280046=40, srv1481069114=21, srv261663032=51, srv646881806=77, srv390581881=56, srv1266703440=8, srv957372773=95, srv2018496474=41, srv3538548=54, srv1852882665=34, srv1293399079=9, srv910626609=91, srv584025241=70, srv463575785=63, srv503016794=68, srv645833509=76, srv1383120285=12, srv1081629712=3, srv1402276609=14, srv1711097026=30, srv398817152=57, srv809457258=86, srv453415199=62, srv70431370=78, srv203038009=42, srv1317181188=10, srv2069260113=46, srv1158771191=5, srv1231189006=7, srv1443647901=19, srv2070005649=47, srv429628147=60, srv619051312=73, srv1049013604=1, srv730765455=80, srv1405271699=16, srv600724505=71, srv240830518=50, srv418569539=59, srv977064743=99, srv371000685=55, srv1858142486=35, srv480282557=66, srv809702192=87, srv115955428=6, srv1442181191=18, srv1067232347=2, srv140516759=15, srv1455855722=20, srv1533106987=24, srv550975866=69, srv1685157820=28, srv91472899=92, srv2067757456=45, srv306981408=53, srv171778577=31, srv1346380814=11, srv817709116=88} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-20 22:53:29,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-20 22:53:29,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-20 22:53:29,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-20 22:53:29,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-20 22:53:29,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-20 22:53:29,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-20 22:53:29,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-20 22:53:29,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-20 22:53:29,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-20 22:53:29,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-20 22:53:29,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-20 22:53:29,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-20 22:53:29,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-20 22:53:29,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-20 22:53:29,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-20 22:53:29,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-20 22:53:29,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-20 22:53:29,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-20 22:53:29,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-20 22:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-20 22:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-20 22:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-20 22:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-20 22:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-20 22:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-20 22:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-20 22:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-20 22:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-20 22:53:29,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-20 22:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-20 22:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-20 22:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-20 22:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-20 22:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-20 22:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-20 22:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-20 22:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-20 22:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-20 22:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-20 22:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-20 22:53:29,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-20 22:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-20 22:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-20 22:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-20 22:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-20 22:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-20 22:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-20 22:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-20 22:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-20 22:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-20 22:53:29,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-20 22:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-20 22:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-20 22:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-20 22:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-20 22:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-20 22:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-20 22:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-20 22:53:29,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-20 22:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-20 22:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-20 22:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-20 22:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-20 22:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-20 22:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-20 22:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-20 22:53:29,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-20 22:53:29,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-20 22:53:29,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-20 22:53:29,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-20 22:53:29,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-20 22:53:29,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-20 22:53:29,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-20 22:53:29,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-20 22:53:29,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-20 22:53:29,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-20 22:53:29,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-20 22:53:29,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-20 22:53:29,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-20 22:53:29,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-20 22:53:29,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-20 22:53:29,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-20 22:53:29,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-20 22:53:29,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-20 22:53:29,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-20 22:53:29,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-20 22:53:29,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-20 22:53:29,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-20 22:53:29,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-20 22:53:29,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-20 22:53:29,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-20 22:53:29,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-20 22:53:29,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-20 22:53:29,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-20 22:53:29,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-20 22:53:29,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-20 22:53:29,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-20 22:53:29,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-20 22:53:29,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-20 22:53:29,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-20 22:53:29,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-20 22:53:29,688 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03993510136131179, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.052059708446615185, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03993510136131179, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17941666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24333333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 22:53:29,688 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-20 22:53:29,688 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1838580006072509, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03993510136131179, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.052059708446615185, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03993510136131179, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17941666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24333333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-20 22:53:29,955 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 386 ms to try 5000 different iterations. Found a solution that moves 2616 regions; Going from a computed imbalance of 0.1838580006072509 to a new imbalance of 0.014262618174701543. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02328545137883201); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031179578340629285, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2906666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02328545137883201); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 22:53:29,964 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv240830518:80 , srv550975866:80 , srv622594928:80 , srv860948605:80 , srv1346380814:81 , srv1794757379:81 , srv646881806:81 , srv764490360:81 , srv1106969105:82 , srv1852882665:82 , srv1866694642:82 , srv215949714:82 , srv3538548:82 , srv584025241:82 , srv799656349:82 , srv919504532:82 , srv1858142486:83 , srv2070005649:83 , srv418547670:83 , srv499012744:83 , srv730765455:83 , srv8987161:83 , srv967891314:83 , srv1049013604:84 , srv115955428:84 , srv1293399079:84 , srv1455855722:84 , srv1488261827:84 , srv2069260113:84 , srv466818714:84 , srv503016794:84 , srv809457258:84 , srv1003992243:85 , srv1405271699:85 , srv1407486881:85 , srv1483642240:85 , srv1865654517:85 , srv262537214:85 , srv619051312:85 , srv622249421:85 , srv645833509:85 , srv706995183:85 , srv750352678:85 , srv791831348:85 , srv817709116:85 , srv963026301:85 , srv1231189006:86 , srv1443647901:86 , srv1481069114:86 , srv1882416371:86 , srv2018496474:86 , srv203038009:86 , srv2038360206:86 , srv2090088428:86 , srv371000685:86 , srv453415199:86 , srv463575785:86 , srv70431370:86 , srv910626609:86 , srv91472899:86 , srv947917814:86 , srv1266703440:87 , srv1383120285:87 , srv1396438687:87 , srv1402276609:87 , srv1442181191:87 , srv1562717304:87 , srv1685157820:87 , srv1711097026:87 , srv2067757456:87 , srv261663032:87 , srv390581881:87 , srv611578354:87 , srv97514374:87 , srv977064743:87 , srv1067232347:88 , srv1081629712:88 , srv1158771191:88 , srv1317181188:88 , srv140516759:88 , srv1533106987:88 , srv1961258881:88 , srv2005280046:88 , srv398817152:88 , srv418569539:88 , srv600724505:88 , srv792262031:88 , srv809702192:88 , srv1573530059:89 , srv171778577:89 , srv1764695934:89 , srv429628147:89 , srv452417545:89 , srv467479739:89 , srv1624934652:90 , srv1705150531:90 , srv480282557:90 , srv957372773:90 , srv2054429522:91 , srv306981408:536 } 2023-10-20 22:53:29,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-20 22:53:29,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1961258881=39, srv622594928=75, srv791831348=83, srv1882416371=38, srv750352678=81, srv8987161=90, srv764490360=82, srv1407486881=17, srv611578354=72, srv622249421=74, srv1488261827=23, srv967891314=97, srv467479739=65, srv97514374=98, srv1624934652=27, srv262537214=52, srv1483642240=22, srv706995183=79, srv1003992243=0, srv860948605=89, srv2054429522=44, srv1764695934=32, srv1865654517=36, srv1866694642=37, srv1106969105=4, srv499012744=67, srv1794757379=33, srv947917814=94, srv799656349=85, srv919504532=93, srv963026301=96, srv215949714=49, srv1573530059=26, srv1705150531=29, srv452417545=61, srv418547670=58, srv1396438687=13, srv2038360206=43, srv2090088428=48, srv1562717304=25, srv466818714=64, srv792262031=84, srv2005280046=40, srv1481069114=21, srv261663032=51, srv646881806=77, srv390581881=56, srv1266703440=8, srv957372773=95, srv2018496474=41, srv3538548=54, srv1852882665=34, srv1293399079=9, srv910626609=91, srv584025241=70, srv463575785=63, srv503016794=68, srv645833509=76, srv1383120285=12, srv1081629712=3, srv1402276609=14, srv1711097026=30, srv398817152=57, srv809457258=86, srv453415199=62, srv70431370=78, srv203038009=42, srv1317181188=10, srv2069260113=46, srv1158771191=5, srv1231189006=7, srv1443647901=19, srv2070005649=47, srv429628147=60, srv619051312=73, srv1049013604=1, srv730765455=80, srv1405271699=16, srv600724505=71, srv240830518=50, srv418569539=59, srv977064743=99, srv371000685=55, srv1858142486=35, srv480282557=66, srv809702192=87, srv115955428=6, srv1442181191=18, srv1067232347=2, srv140516759=15, srv1455855722=20, srv1533106987=24, srv550975866=69, srv1685157820=28, srv91472899=92, srv2067757456=45, srv306981408=53, srv171778577=31, srv1346380814=11, srv817709116=88} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-20 22:53:29,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-20 22:53:29,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-20 22:53:29,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-20 22:53:29,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-20 22:53:29,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-20 22:53:29,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-20 22:53:29,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-20 22:53:29,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-20 22:53:29,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-20 22:53:29,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-20 22:53:29,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-20 22:53:29,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-20 22:53:29,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-20 22:53:29,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-20 22:53:29,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-20 22:53:29,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-20 22:53:29,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-20 22:53:29,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-20 22:53:29,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-20 22:53:29,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-20 22:53:29,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-20 22:53:29,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-20 22:53:29,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-20 22:53:29,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-20 22:53:29,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-20 22:53:29,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-20 22:53:29,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-20 22:53:29,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-20 22:53:29,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-20 22:53:29,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-20 22:53:29,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-20 22:53:29,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-20 22:53:29,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-20 22:53:29,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-20 22:53:29,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-20 22:53:29,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-20 22:53:29,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-20 22:53:29,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-20 22:53:29,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-20 22:53:29,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-20 22:53:29,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-20 22:53:29,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-20 22:53:29,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-20 22:53:29,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-20 22:53:29,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-20 22:53:29,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-20 22:53:29,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-20 22:53:29,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-20 22:53:29,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-20 22:53:29,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-20 22:53:29,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-20 22:53:29,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-20 22:53:29,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-20 22:53:29,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-20 22:53:29,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-20 22:53:29,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-20 22:53:29,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-20 22:53:29,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-20 22:53:29,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-20 22:53:29,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-20 22:53:29,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-20 22:53:29,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-20 22:53:29,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-20 22:53:29,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-20 22:53:29,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-20 22:53:29,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-20 22:53:29,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-20 22:53:29,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-20 22:53:29,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-20 22:53:29,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-20 22:53:29,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-20 22:53:29,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-20 22:53:29,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-20 22:53:29,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-20 22:53:29,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-20 22:53:29,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-20 22:53:29,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-20 22:53:29,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-20 22:53:29,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-20 22:53:29,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-20 22:53:29,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-20 22:53:29,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-20 22:53:29,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-20 22:53:29,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-20 22:53:29,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-20 22:53:29,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-20 22:53:29,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-20 22:53:29,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-20 22:53:29,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-20 22:53:29,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-20 22:53:29,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-20 22:53:29,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-20 22:53:29,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-20 22:53:29,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-20 22:53:29,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-20 22:53:29,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-20 22:53:29,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-20 22:53:29,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-20 22:53:29,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-20 22:53:29,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-20 22:53:29,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-20 22:53:30,103 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02328545137883201); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031179578340629285, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02328545137883201); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 22:53:30,104 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-20 22:53:30,104 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014244300551191125, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02328545137883201); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031179578340629285, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02328545137883201); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-20 22:53:30,380 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 375 ms to try 5000 different iterations. Found a solution that moves 967 regions; Going from a computed imbalance of 0.014244300551191125 to a new imbalance of 8.334661726074241E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008004550395605613); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00842208629724276); MoveCostFunction : (multiplier=7.0, imbalance=0.10744444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008004550395605613); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 22:53:30,382 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1533106987:82 , srv1067232347:83 , srv1158771191:83 , srv2005280046:83 , srv418547670:83 , srv418569539:83 , srv429628147:83 , srv619051312:83 , srv622594928:83 , srv764490360:83 , srv1049013604:84 , srv1106969105:84 , srv115955428:84 , srv1231189006:84 , srv1481069114:84 , srv171778577:84 , srv203038009:84 , srv3538548:84 , srv371000685:84 , srv398817152:84 , srv452417545:84 , srv467479739:84 , srv480282557:84 , srv809702192:84 , srv860948605:84 , srv91472899:84 , srv919504532:84 , srv1003992243:85 , srv1081629712:85 , srv1317181188:85 , srv1396438687:85 , srv140516759:85 , srv1405271699:85 , srv1455855722:85 , srv1562717304:85 , srv1624934652:85 , srv1961258881:85 , srv2018496474:85 , srv2067757456:85 , srv2069260113:85 , srv261663032:85 , srv390581881:85 , srv503016794:85 , srv600724505:85 , srv622249421:85 , srv70431370:85 , srv791831348:85 , srv792262031:85 , srv957372773:85 , srv97514374:85 , srv977064743:85 , srv1266703440:86 , srv1293399079:86 , srv1346380814:86 , srv1383120285:86 , srv1402276609:86 , srv1685157820:86 , srv1705150531:86 , srv1711097026:86 , srv1764695934:86 , srv1852882665:86 , srv1865654517:86 , srv1882416371:86 , srv2038360206:86 , srv2054429522:86 , srv2070005649:86 , srv2090088428:86 , srv262537214:86 , srv453415199:86 , srv499012744:86 , srv550975866:86 , srv611578354:86 , srv706995183:86 , srv799656349:86 , srv809457258:86 , srv910626609:86 , srv947917814:86 , srv967891314:86 , srv1442181191:87 , srv1443647901:87 , srv1483642240:87 , srv1488261827:87 , srv1794757379:87 , srv1858142486:87 , srv1866694642:87 , srv463575785:87 , srv466818714:87 , srv584025241:87 , srv645833509:87 , srv646881806:87 , srv730765455:87 , srv817709116:87 , srv8987161:87 , srv963026301:87 , srv1407486881:88 , srv1573530059:88 , srv215949714:88 , srv750352678:88 , srv240830518:89 , srv306981408:548 } 2023-10-20 22:53:30,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-20 22:53:30,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1961258881=39, srv622594928=75, srv791831348=83, srv1882416371=38, srv750352678=81, srv8987161=90, srv764490360=82, srv1407486881=17, srv611578354=72, srv622249421=74, srv1488261827=23, srv967891314=97, srv467479739=65, srv97514374=98, srv1624934652=27, srv262537214=52, srv1483642240=22, srv706995183=79, srv1003992243=0, srv860948605=89, srv2054429522=44, srv1764695934=32, srv1865654517=36, srv1866694642=37, srv1106969105=4, srv499012744=67, srv1794757379=33, srv947917814=94, srv799656349=85, srv919504532=93, srv963026301=96, srv215949714=49, srv1573530059=26, srv1705150531=29, srv452417545=61, srv418547670=58, srv1396438687=13, srv2038360206=43, srv2090088428=48, srv1562717304=25, srv466818714=64, srv792262031=84, srv2005280046=40, srv1481069114=21, srv261663032=51, srv646881806=77, srv390581881=56, srv1266703440=8, srv957372773=95, srv2018496474=41, srv3538548=54, srv1852882665=34, srv1293399079=9, srv910626609=91, srv584025241=70, srv463575785=63, srv503016794=68, srv645833509=76, srv1383120285=12, srv1081629712=3, srv1402276609=14, srv1711097026=30, srv398817152=57, srv809457258=86, srv453415199=62, srv70431370=78, srv203038009=42, srv1317181188=10, srv2069260113=46, srv1158771191=5, srv1231189006=7, srv1443647901=19, srv2070005649=47, srv429628147=60, srv619051312=73, srv1049013604=1, srv730765455=80, srv1405271699=16, srv600724505=71, srv240830518=50, srv418569539=59, srv977064743=99, srv371000685=55, srv1858142486=35, srv480282557=66, srv809702192=87, srv115955428=6, srv1442181191=18, srv1067232347=2, srv140516759=15, srv1455855722=20, srv1533106987=24, srv550975866=69, srv1685157820=28, srv91472899=92, srv2067757456=45, srv306981408=53, srv171778577=31, srv1346380814=11, srv817709116=88} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-20 22:53:30,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-20 22:53:30,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-20 22:53:30,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-20 22:53:30,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-20 22:53:30,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-20 22:53:30,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-20 22:53:30,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-20 22:53:30,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-20 22:53:30,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-20 22:53:30,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-20 22:53:30,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-20 22:53:30,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-20 22:53:30,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-20 22:53:30,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-20 22:53:30,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-20 22:53:30,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-20 22:53:30,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-20 22:53:30,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-20 22:53:30,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-20 22:53:30,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-20 22:53:30,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-20 22:53:30,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-20 22:53:30,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-20 22:53:30,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-20 22:53:30,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-20 22:53:30,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-20 22:53:30,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-20 22:53:30,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-20 22:53:30,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-20 22:53:30,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-20 22:53:30,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-20 22:53:30,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-20 22:53:30,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-20 22:53:30,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-20 22:53:30,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-20 22:53:30,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-20 22:53:30,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-20 22:53:30,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-20 22:53:30,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-20 22:53:30,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-20 22:53:30,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-20 22:53:30,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-20 22:53:30,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-20 22:53:30,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-20 22:53:30,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-20 22:53:30,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-20 22:53:30,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-20 22:53:30,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-20 22:53:30,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-20 22:53:30,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-20 22:53:30,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-20 22:53:30,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-20 22:53:30,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-20 22:53:30,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-20 22:53:30,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-20 22:53:30,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-20 22:53:30,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-20 22:53:30,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-20 22:53:30,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-20 22:53:30,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-20 22:53:30,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-20 22:53:30,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-20 22:53:30,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-20 22:53:30,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-20 22:53:30,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-20 22:53:30,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-20 22:53:30,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-20 22:53:30,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-20 22:53:30,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-20 22:53:30,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-20 22:53:30,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-20 22:53:30,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-20 22:53:30,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-20 22:53:30,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-20 22:53:30,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-20 22:53:30,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-20 22:53:30,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-20 22:53:30,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-20 22:53:30,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-20 22:53:30,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-20 22:53:30,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-20 22:53:30,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-20 22:53:30,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-20 22:53:30,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-20 22:53:30,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-20 22:53:30,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-20 22:53:30,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-20 22:53:30,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-20 22:53:30,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-20 22:53:30,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-20 22:53:30,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-20 22:53:30,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-20 22:53:30,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-20 22:53:30,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-20 22:53:30,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-20 22:53:30,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-20 22:53:30,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-20 22:53:30,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-20 22:53:30,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-20 22:53:30,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-20 22:53:30,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-20 22:53:30,509 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008004550395605613); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00842208629724276); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008004550395605613); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 22:53:30,509 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-20 22:53:30,510 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.266950938862565E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008004550395605613); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00842208629724276); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008004550395605613); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-20 22:53:30,862 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 448 ms to try 5000 different iterations. Found a solution that moves 270 regions; Going from a computed imbalance of 8.266950938862565E-4 to a new imbalance of 5.686502741935342E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.03, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 22:53:30,863 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1573530059:83 , srv2067757456:83 , srv480282557:83 , srv977064743:83 , srv1158771191:84 , srv1231189006:84 , srv1293399079:84 , srv1317181188:84 , srv1383120285:84 , srv140516759:84 , srv1407486881:84 , srv1443647901:84 , srv1481069114:84 , srv1705150531:84 , srv1711097026:84 , srv1852882665:84 , srv203038009:84 , srv2038360206:84 , srv215949714:84 , srv240830518:84 , srv261663032:84 , srv371000685:84 , srv398817152:84 , srv452417545:84 , srv453415199:84 , srv463575785:84 , srv600724505:84 , srv611578354:84 , srv70431370:84 , srv706995183:84 , srv730765455:84 , srv750352678:84 , srv792262031:84 , srv817709116:84 , srv91472899:84 , srv957372773:84 , srv97514374:84 , srv1003992243:85 , srv1049013604:85 , srv1081629712:85 , srv1106969105:85 , srv1266703440:85 , srv1396438687:85 , srv1402276609:85 , srv1405271699:85 , srv1442181191:85 , srv1455855722:85 , srv1483642240:85 , srv1533106987:85 , srv1562717304:85 , srv1624934652:85 , srv1685157820:85 , srv171778577:85 , srv1764695934:85 , srv1794757379:85 , srv1858142486:85 , srv1865654517:85 , srv1866694642:85 , srv1882416371:85 , srv1961258881:85 , srv2005280046:85 , srv2018496474:85 , srv2054429522:85 , srv2070005649:85 , srv2090088428:85 , srv262537214:85 , srv3538548:85 , srv418569539:85 , srv429628147:85 , srv466818714:85 , srv467479739:85 , srv499012744:85 , srv503016794:85 , srv550975866:85 , srv584025241:85 , srv619051312:85 , srv622249421:85 , srv645833509:85 , srv646881806:85 , srv791831348:85 , srv809457258:85 , srv860948605:85 , srv8987161:85 , srv910626609:85 , srv919504532:85 , srv963026301:85 , srv967891314:85 , srv1067232347:86 , srv115955428:86 , srv1346380814:86 , srv1488261827:86 , srv390581881:86 , srv418547670:86 , srv622594928:86 , srv764490360:86 , srv799656349:86 , srv809702192:86 , srv947917814:86 , srv2069260113:87 , srv306981408:613 } 2023-10-20 22:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-20 22:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1961258881=39, srv622594928=75, srv791831348=83, srv1882416371=38, srv750352678=81, srv8987161=90, srv764490360=82, srv1407486881=17, srv611578354=72, srv622249421=74, srv1488261827=23, srv967891314=97, srv467479739=65, srv97514374=98, srv1624934652=27, srv262537214=52, srv1483642240=22, srv706995183=79, srv1003992243=0, srv860948605=89, srv2054429522=44, srv1764695934=32, srv1865654517=36, srv1866694642=37, srv1106969105=4, srv499012744=67, srv1794757379=33, srv947917814=94, srv799656349=85, srv919504532=93, srv963026301=96, srv215949714=49, srv1573530059=26, srv1705150531=29, srv452417545=61, srv418547670=58, srv1396438687=13, srv2038360206=43, srv2090088428=48, srv1562717304=25, srv466818714=64, srv792262031=84, srv2005280046=40, srv1481069114=21, srv261663032=51, srv646881806=77, srv390581881=56, srv1266703440=8, srv957372773=95, srv2018496474=41, srv3538548=54, srv1852882665=34, srv1293399079=9, srv910626609=91, srv584025241=70, srv463575785=63, srv503016794=68, srv645833509=76, srv1383120285=12, srv1081629712=3, srv1402276609=14, srv1711097026=30, srv398817152=57, srv809457258=86, srv453415199=62, srv70431370=78, srv203038009=42, srv1317181188=10, srv2069260113=46, srv1158771191=5, srv1231189006=7, srv1443647901=19, srv2070005649=47, srv429628147=60, srv619051312=73, srv1049013604=1, srv730765455=80, srv1405271699=16, srv600724505=71, srv240830518=50, srv418569539=59, srv977064743=99, srv371000685=55, srv1858142486=35, srv480282557=66, srv809702192=87, srv115955428=6, srv1442181191=18, srv1067232347=2, srv140516759=15, srv1455855722=20, srv1533106987=24, srv550975866=69, srv1685157820=28, srv91472899=92, srv2067757456=45, srv306981408=53, srv171778577=31, srv1346380814=11, srv817709116=88} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-20 22:53:30,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-20 22:53:30,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-20 22:53:30,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-20 22:53:30,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-20 22:53:30,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-20 22:53:30,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-20 22:53:30,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-20 22:53:30,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-20 22:53:30,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-20 22:53:30,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-20 22:53:30,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-20 22:53:30,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-20 22:53:30,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-20 22:53:30,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-20 22:53:30,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-20 22:53:30,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-20 22:53:30,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-20 22:53:30,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-20 22:53:30,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-20 22:53:30,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-20 22:53:30,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-20 22:53:30,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-20 22:53:30,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-20 22:53:30,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-20 22:53:30,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-20 22:53:30,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-20 22:53:30,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-20 22:53:30,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-20 22:53:30,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-20 22:53:30,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-20 22:53:30,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-20 22:53:30,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-20 22:53:30,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-20 22:53:30,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-20 22:53:30,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-20 22:53:30,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-20 22:53:30,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-20 22:53:30,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-20 22:53:30,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-20 22:53:30,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-20 22:53:30,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-20 22:53:30,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-20 22:53:30,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-20 22:53:30,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-20 22:53:30,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-20 22:53:30,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-20 22:53:30,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-20 22:53:30,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-20 22:53:30,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-20 22:53:30,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-20 22:53:30,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-20 22:53:30,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-20 22:53:30,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-20 22:53:30,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-20 22:53:30,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-20 22:53:30,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-20 22:53:30,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-20 22:53:30,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-20 22:53:30,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-20 22:53:30,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-20 22:53:30,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-20 22:53:30,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-20 22:53:30,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-20 22:53:30,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-20 22:53:30,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-20 22:53:30,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-20 22:53:30,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-20 22:53:30,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-20 22:53:30,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-20 22:53:30,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-20 22:53:30,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-20 22:53:30,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-20 22:53:30,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-20 22:53:30,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-20 22:53:30,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-20 22:53:30,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-20 22:53:30,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-20 22:53:30,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-20 22:53:30,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-20 22:53:30,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-20 22:53:30,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-20 22:53:30,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-20 22:53:30,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-20 22:53:30,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-20 22:53:30,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-20 22:53:30,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-20 22:53:30,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-20 22:53:30,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-20 22:53:30,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-20 22:53:30,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-20 22:53:30,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-20 22:53:30,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-20 22:53:30,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-20 22:53:30,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-20 22:53:30,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-20 22:53:30,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-20 22:53:30,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-20 22:53:30,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-20 22:53:30,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-20 22:53:30,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-20 22:53:30,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-20 22:53:30,990 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 22:53:30,991 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-20 22:53:30,991 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.4974447011168104E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-20 22:53:31,235 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 344 ms to try 5000 different iterations. Found a solution that moves 26 regions; Going from a computed imbalance of 5.4974447011168104E-5 to a new imbalance of 7.569813391471145E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0028888888888888888); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 22:53:31,236 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1488261827:83 , srv2069260113:83 , srv390581881:83 , srv466818714:83 , srv764490360:83 , srv1003992243:84 , srv1049013604:84 , srv1067232347:84 , srv1081629712:84 , srv1106969105:84 , srv1158771191:84 , srv115955428:84 , srv1231189006:84 , srv1266703440:84 , srv1346380814:84 , srv1383120285:84 , srv1396438687:84 , srv1402276609:84 , srv140516759:84 , srv1405271699:84 , srv1407486881:84 , srv1442181191:84 , srv1443647901:84 , srv1455855722:84 , srv1481069114:84 , srv1483642240:84 , srv1533106987:84 , srv1562717304:84 , srv1573530059:84 , srv1624934652:84 , srv1685157820:84 , srv171778577:84 , srv1764695934:84 , srv1794757379:84 , srv1852882665:84 , srv1858142486:84 , srv1865654517:84 , srv1866694642:84 , srv1882416371:84 , srv1961258881:84 , srv2005280046:84 , srv2018496474:84 , srv203038009:84 , srv2038360206:84 , srv2054429522:84 , srv2067757456:84 , srv2070005649:84 , srv2090088428:84 , srv215949714:84 , srv240830518:84 , srv261663032:84 , srv262537214:84 , srv3538548:84 , srv371000685:84 , srv418547670:84 , srv418569539:84 , srv429628147:84 , srv452417545:84 , srv453415199:84 , srv463575785:84 , srv467479739:84 , srv480282557:84 , srv499012744:84 , srv503016794:84 , srv550975866:84 , srv584025241:84 , srv600724505:84 , srv611578354:84 , srv619051312:84 , srv622249421:84 , srv622594928:84 , srv645833509:84 , srv646881806:84 , srv70431370:84 , srv706995183:84 , srv730765455:84 , srv750352678:84 , srv791831348:84 , srv792262031:84 , srv799656349:84 , srv809457258:84 , srv809702192:84 , srv817709116:84 , srv860948605:84 , srv8987161:84 , srv910626609:84 , srv91472899:84 , srv919504532:84 , srv947917814:84 , srv957372773:84 , srv963026301:84 , srv967891314:84 , srv97514374:84 , srv977064743:84 , srv1293399079:85 , srv1317181188:85 , srv1705150531:85 , srv1711097026:85 , srv398817152:85 , srv306981408:684 } 2023-10-20 22:53:31,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-20 22:53:31,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1961258881=39, srv622594928=75, srv791831348=83, srv1882416371=38, srv750352678=81, srv8987161=90, srv764490360=82, srv1407486881=17, srv611578354=72, srv622249421=74, srv1488261827=23, srv967891314=97, srv467479739=65, srv97514374=98, srv1624934652=27, srv262537214=52, srv1483642240=22, srv706995183=79, srv1003992243=0, srv860948605=89, srv2054429522=44, srv1764695934=32, srv1865654517=36, srv1866694642=37, srv1106969105=4, srv499012744=67, srv1794757379=33, srv947917814=94, srv799656349=85, srv919504532=93, srv963026301=96, srv215949714=49, srv1573530059=26, srv1705150531=29, srv452417545=61, srv418547670=58, srv1396438687=13, srv2038360206=43, srv2090088428=48, srv1562717304=25, srv466818714=64, srv792262031=84, srv2005280046=40, srv1481069114=21, srv261663032=51, srv646881806=77, srv390581881=56, srv1266703440=8, srv957372773=95, srv2018496474=41, srv3538548=54, srv1852882665=34, srv1293399079=9, srv910626609=91, srv584025241=70, srv463575785=63, srv503016794=68, srv645833509=76, srv1383120285=12, srv1081629712=3, srv1402276609=14, srv1711097026=30, srv398817152=57, srv809457258=86, srv453415199=62, srv70431370=78, srv203038009=42, srv1317181188=10, srv2069260113=46, srv1158771191=5, srv1231189006=7, srv1443647901=19, srv2070005649=47, srv429628147=60, srv619051312=73, srv1049013604=1, srv730765455=80, srv1405271699=16, srv600724505=71, srv240830518=50, srv418569539=59, srv977064743=99, srv371000685=55, srv1858142486=35, srv480282557=66, srv809702192=87, srv115955428=6, srv1442181191=18, srv1067232347=2, srv140516759=15, srv1455855722=20, srv1533106987=24, srv550975866=69, srv1685157820=28, srv91472899=92, srv2067757456=45, srv306981408=53, srv171778577=31, srv1346380814=11, srv817709116=88} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-20 22:53:31,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-20 22:53:31,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-20 22:53:31,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-20 22:53:31,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-20 22:53:31,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-20 22:53:31,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-20 22:53:31,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-20 22:53:31,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-20 22:53:31,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-20 22:53:31,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-20 22:53:31,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-20 22:53:31,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-20 22:53:31,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-20 22:53:31,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-20 22:53:31,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-20 22:53:31,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-20 22:53:31,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-20 22:53:31,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-20 22:53:31,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-20 22:53:31,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-20 22:53:31,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-20 22:53:31,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-20 22:53:31,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-20 22:53:31,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-20 22:53:31,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-20 22:53:31,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-20 22:53:31,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-20 22:53:31,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-20 22:53:31,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-20 22:53:31,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-20 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-20 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-20 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-20 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-20 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-20 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-20 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-20 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-20 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-20 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-20 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-20 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-20 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-20 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-20 22:53:31,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-20 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-20 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-20 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-20 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-20 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-20 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-20 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-20 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-20 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-20 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-20 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-20 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-20 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-20 22:53:31,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-20 22:53:31,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-20 22:53:31,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-20 22:53:31,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-20 22:53:31,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-20 22:53:31,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-20 22:53:31,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-20 22:53:31,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-20 22:53:31,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-20 22:53:31,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-20 22:53:31,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-20 22:53:31,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-20 22:53:31,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-20 22:53:31,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-20 22:53:31,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-20 22:53:31,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-20 22:53:31,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-20 22:53:31,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-20 22:53:31,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-20 22:53:31,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-20 22:53:31,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-20 22:53:31,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-20 22:53:31,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-20 22:53:31,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-20 22:53:31,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-20 22:53:31,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-20 22:53:31,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-20 22:53:31,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-20 22:53:31,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-20 22:53:31,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-20 22:53:31,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-20 22:53:31,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-20 22:53:31,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-20 22:53:31,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-20 22:53:31,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-20 22:53:31,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-20 22:53:31,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-20 22:53:31,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-20 22:53:31,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-20 22:53:31,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-20 22:53:31,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-20 22:53:31,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-20 22:53:31,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-20 22:53:31,387 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 22:53:31,387 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1488261827:83 , srv2069260113:83 , srv390581881:83 , srv466818714:83 , srv764490360:83 , srv1003992243:84 , srv1049013604:84 , srv1067232347:84 , srv1081629712:84 , srv1106969105:84 , srv1158771191:84 , srv115955428:84 , srv1231189006:84 , srv1266703440:84 , srv1346380814:84 , srv1383120285:84 , srv1396438687:84 , srv1402276609:84 , srv140516759:84 , srv1405271699:84 , srv1407486881:84 , srv1442181191:84 , srv1443647901:84 , srv1455855722:84 , srv1481069114:84 , srv1483642240:84 , srv1533106987:84 , srv1562717304:84 , srv1573530059:84 , srv1624934652:84 , srv1685157820:84 , srv171778577:84 , srv1764695934:84 , srv1794757379:84 , srv1852882665:84 , srv1858142486:84 , srv1865654517:84 , srv1866694642:84 , srv1882416371:84 , srv1961258881:84 , srv2005280046:84 , srv2018496474:84 , srv203038009:84 , srv2038360206:84 , srv2054429522:84 , srv2067757456:84 , srv2070005649:84 , srv2090088428:84 , srv215949714:84 , srv240830518:84 , srv261663032:84 , srv262537214:84 , srv3538548:84 , srv371000685:84 , srv418547670:84 , srv418569539:84 , srv429628147:84 , srv452417545:84 , srv453415199:84 , srv463575785:84 , srv467479739:84 , srv480282557:84 , srv499012744:84 , srv503016794:84 , srv550975866:84 , srv584025241:84 , srv600724505:84 , srv611578354:84 , srv619051312:84 , srv622249421:84 , srv622594928:84 , srv645833509:84 , srv646881806:84 , srv70431370:84 , srv706995183:84 , srv730765455:84 , srv750352678:84 , srv791831348:84 , srv792262031:84 , srv799656349:84 , srv809457258:84 , srv809702192:84 , srv817709116:84 , srv860948605:84 , srv8987161:84 , srv910626609:84 , srv91472899:84 , srv919504532:84 , srv947917814:84 , srv957372773:84 , srv963026301:84 , srv967891314:84 , srv97514374:84 , srv977064743:84 , srv1293399079:85 , srv1317181188:85 , srv1705150531:85 , srv1711097026:85 , srv398817152:85 , srv306981408:684 } 2023-10-20 22:53:31,413 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=360 (was 357) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=8662 (was 9185) 2023-10-20 22:53:31,420 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=360, ProcessCount=169, AvailableMemoryMB=8662 2023-10-20 22:53:31,421 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-20 22:53:31,421 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-10-20 22:53:31,433 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1145838939:84 , srv1146088020:84 , srv1161707771:84 , srv129885224:84 , srv1327536613:84 , srv1330505758:84 , srv143843499:84 , srv1501037257:84 , srv1510962237:84 , srv1516357984:84 , srv161718505:84 , srv1623971629:84 , srv1687682446:84 , srv1715042136:84 , srv1732962738:84 , srv1974241377:84 , srv1997753572:84 , srv2057416375:84 , srv2078894254:84 , srv2088652073:84 , srv265334831:84 , srv435406581:84 , srv527765881:84 , srv565671432:84 , srv566262535:84 , srv687576207:84 , srv774791440:84 , srv989507595:84 , srv999248351:84 , srv926718353:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-10-20 22:53:31,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-20 22:53:31,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1327536613=4, srv2057416375=17, srv1687682446=12, srv1510962237=8, srv1732962738=14, srv1330505758=5, srv1516357984=9, srv774791440=26, srv1145838939=0, srv129885224=3, srv1161707771=2, srv435406581=21, srv989507595=28, srv1974241377=15, srv527765881=22, srv2088652073=19, srv566262535=24, srv143843499=6, srv161718505=10, srv1715042136=13, srv2078894254=18, srv265334831=20, srv1501037257=7, srv999248351=29, srv565671432=23, srv926718353=27, srv1146088020=1, srv687576207=25, srv1997753572=16, srv1623971629=11} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-20 22:53:31,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-20 22:53:31,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-20 22:53:31,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-20 22:53:31,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-20 22:53:31,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-20 22:53:31,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-20 22:53:31,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-20 22:53:31,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-20 22:53:31,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-20 22:53:31,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-20 22:53:31,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-20 22:53:31,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-20 22:53:31,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-20 22:53:31,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-20 22:53:31,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-20 22:53:31,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-20 22:53:31,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-20 22:53:31,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-20 22:53:31,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-20 22:53:31,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-20 22:53:31,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-20 22:53:31,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-20 22:53:31,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-20 22:53:31,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-20 22:53:31,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-20 22:53:31,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-20 22:53:31,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-20 22:53:31,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-20 22:53:31,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-20 22:53:31,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-20 22:53:31,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-20 22:53:31,475 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1915392557441424, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 22:53:31,475 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912796577204797, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1915392557441424, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-10-20 22:55:31,455 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5562777 different iterations. Found a solution that moves 1866 regions; Going from a computed imbalance of 0.9912796577204797 to a new imbalance of 4.3553370884861656E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6911111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 22:55:31,463 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1145838939:90 , srv1146088020:90 , srv1161707771:90 , srv129885224:90 , srv1327536613:90 , srv1330505758:90 , srv143843499:90 , srv1501037257:90 , srv1510962237:90 , srv1516357984:90 , srv161718505:90 , srv1623971629:90 , srv1687682446:90 , srv1715042136:90 , srv1732962738:90 , srv1974241377:90 , srv1997753572:90 , srv2057416375:90 , srv2078894254:90 , srv2088652073:90 , srv265334831:90 , srv435406581:90 , srv527765881:90 , srv565671432:90 , srv566262535:90 , srv687576207:90 , srv774791440:90 , srv926718353:90 , srv989507595:90 , srv999248351:90 } 2023-10-20 22:55:31,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-20 22:55:31,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1327536613=4, srv2057416375=17, srv1687682446=12, srv1510962237=8, srv1732962738=14, srv1330505758=5, srv1516357984=9, srv774791440=26, srv1145838939=0, srv129885224=3, srv1161707771=2, srv435406581=21, srv989507595=28, srv1974241377=15, srv527765881=22, srv2088652073=19, srv566262535=24, srv143843499=6, srv161718505=10, srv1715042136=13, srv2078894254=18, srv265334831=20, srv1501037257=7, srv999248351=29, srv565671432=23, srv926718353=27, srv1146088020=1, srv687576207=25, srv1997753572=16, srv1623971629=11} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-20 22:55:31,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-20 22:55:31,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-20 22:55:31,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-20 22:55:31,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-20 22:55:31,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-20 22:55:31,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-20 22:55:31,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-20 22:55:31,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-20 22:55:31,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-20 22:55:31,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-20 22:55:31,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-20 22:55:31,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-20 22:55:31,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-20 22:55:31,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-20 22:55:31,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-20 22:55:31,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-20 22:55:31,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-20 22:55:31,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-20 22:55:31,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-20 22:55:31,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-20 22:55:31,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-20 22:55:31,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-20 22:55:31,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-20 22:55:31,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-20 22:55:31,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-20 22:55:31,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-20 22:55:31,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-20 22:55:31,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-20 22:55:31,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-20 22:55:31,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-20 22:55:31,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-20 22:55:31,483 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-20 22:55:31,483 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1145838939:90 , srv1146088020:90 , srv1161707771:90 , srv129885224:90 , srv1327536613:90 , srv1330505758:90 , srv143843499:90 , srv1501037257:90 , srv1510962237:90 , srv1516357984:90 , srv161718505:90 , srv1623971629:90 , srv1687682446:90 , srv1715042136:90 , srv1732962738:90 , srv1974241377:90 , srv1997753572:90 , srv2057416375:90 , srv2078894254:90 , srv2088652073:90 , srv265334831:90 , srv435406581:90 , srv527765881:90 , srv565671432:90 , srv566262535:90 , srv687576207:90 , srv774791440:90 , srv926718353:90 , srv989507595:90 , srv999248351:90 } 2023-10-20 22:55:31,496 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=234 (was 360), ProcessCount=169 (was 169), AvailableMemoryMB=8655 (was 8662)