2023-08-24 13:58:00,397 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-08-24 13:58:00,722 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-08-24 13:58:00,840 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-08-24 13:58:00,904 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-24 13:58:00,914 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-24 13:58:00,929 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=303, ProcessCount=172, AvailableMemoryMB=11582 2023-08-24 13:58:00,931 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-24 13:58:00,931 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-24 13:58:01,065 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1001604145:84 , srv1004016818:84 , srv1018285316:84 , srv106941380:84 , srv1076099564:84 , srv1089509270:84 , srv1139509108:84 , srv115543251:84 , srv1157760957:84 , srv1178261019:84 , srv1212590300:84 , srv123135318:84 , srv1246818358:84 , srv1274399778:84 , srv1284185606:84 , srv1291973037:84 , srv1297992494:84 , srv1299000668:84 , srv1318634086:84 , srv1327636093:84 , srv1328827719:84 , srv1330788157:84 , srv134364819:84 , srv1364436596:84 , srv1367992234:84 , srv137615500:84 , srv1382031616:84 , srv1405332683:84 , srv1407730602:84 , srv1434237857:84 , srv1451012082:84 , srv145341218:84 , srv1463792196:84 , srv1522232073:84 , srv1544006684:84 , srv1583419286:84 , srv1615643684:84 , srv1691994868:84 , srv1703334845:84 , srv1744179849:84 , srv1748596458:84 , srv1754711791:84 , srv1768123917:84 , srv180278264:84 , srv1827213846:84 , srv1833649697:84 , srv1873148042:84 , srv1876257834:84 , srv1899698486:84 , srv1918082644:84 , srv1934731585:84 , srv1949212252:84 , srv1999459271:84 , srv2020782619:84 , srv2028630977:84 , srv2034479452:84 , srv2036324752:84 , srv2048038018:84 , srv2064444310:84 , srv2075193292:84 , srv208702260:84 , srv209550640:84 , srv227485484:84 , srv232499113:84 , srv237441112:84 , srv316056927:84 , srv316870931:84 , srv336628879:84 , srv380614764:84 , srv384136672:84 , srv443996363:84 , srv491637507:84 , srv512073002:84 , srv550978423:84 , srv554673701:84 , srv559438311:84 , srv582377435:84 , srv589713490:84 , srv615756302:84 , srv646980193:84 , srv657777772:84 , srv694347450:84 , srv709584708:84 , srv721999540:84 , srv741579659:84 , srv751483481:84 , srv757263530:84 , srv771962791:84 , srv791563248:84 , srv855219536:84 , srv872281481:84 , srv897078711:84 , srv915561945:84 , srv95033233:84 , srv967388585:84 , srv969586164:84 , srv979420311:84 , srv995965519:84 , srv99696301:84 , srv103458101:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-08-24 13:58:01,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-24 13:58:01,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1076099564=5, srv1949212252=52, srv2036324752=57, srv550978423=74, srv95033233=94, srv872281481=91, srv915561945=93, srv1297992494=17, srv721999540=84, srv1299000668=18, srv1405332683=28, srv134364819=23, srv709584708=83, srv1463792196=33, srv99696301=99, srv2034479452=56, srv855219536=90, srv657777772=81, srv1212590300=11, srv967388585=95, srv1918082644=50, srv559438311=76, srv1522232073=34, srv646980193=80, srv615756302=79, srv443996363=71, srv1768123917=43, srv145341218=32, srv1407730602=29, srv1364436596=24, srv1833649697=46, srv106941380=4, srv1691994868=38, srv336628879=68, srv1544006684=35, srv115543251=8, srv316870931=67, srv1318634086=19, srv1274399778=14, srv1615643684=37, srv1873148042=47, srv1291973037=16, srv757263530=87, srv554673701=75, srv1876257834=48, srv180278264=44, srv384136672=70, srv1382031616=27, srv995965519=98, srv1001604145=0, srv1754711791=42, srv208702260=61, srv491637507=72, srv1744179849=40, srv137615500=26, srv1899698486=49, srv1748596458=41, srv1330788157=22, srv751483481=86, srv2064444310=59, srv741579659=85, srv791563248=89, srv1451012082=31, srv979420311=97, srv1367992234=25, srv969586164=96, srv1089509270=6, srv237441112=65, srv1139509108=7, srv589713490=78, srv1703334845=39, srv1583419286=36, srv380614764=69, srv1004016818=1, srv2020782619=54, srv1827213846=45, srv897078711=92, srv694347450=82, srv2075193292=60, srv227485484=63, srv1246818358=13, srv1157760957=9, srv209550640=62, srv771962791=88, srv1434237857=30, srv2028630977=55, srv232499113=64, srv2048038018=58, srv1999459271=53, srv512073002=73, srv1178261019=10, srv1934731585=51, srv582377435=77, srv1328827719=21, srv316056927=66, srv103458101=3, srv123135318=12, srv1018285316=2, srv1284185606=15, srv1327636093=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-24 13:58:01,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-24 13:58:01,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-24 13:58:01,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-24 13:58:01,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-24 13:58:01,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-24 13:58:01,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-24 13:58:01,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-24 13:58:01,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-24 13:58:01,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-24 13:58:01,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-24 13:58:01,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-24 13:58:01,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-24 13:58:01,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-24 13:58:01,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-24 13:58:01,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-24 13:58:01,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-24 13:58:01,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-24 13:58:01,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-24 13:58:01,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-24 13:58:01,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-24 13:58:01,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-24 13:58:01,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-24 13:58:01,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-24 13:58:01,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-24 13:58:01,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-24 13:58:01,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-24 13:58:01,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-24 13:58:01,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-24 13:58:01,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-24 13:58:01,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-24 13:58:01,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-24 13:58:01,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-24 13:58:01,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-24 13:58:01,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-24 13:58:01,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-24 13:58:01,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-24 13:58:01,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-24 13:58:01,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-24 13:58:01,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-24 13:58:01,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-24 13:58:01,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-24 13:58:01,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-24 13:58:01,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-24 13:58:01,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-24 13:58:01,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-24 13:58:01,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-24 13:58:01,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-24 13:58:01,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-24 13:58:01,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-24 13:58:01,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-24 13:58:01,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-24 13:58:01,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-24 13:58:01,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-24 13:58:01,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-24 13:58:01,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-24 13:58:01,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-24 13:58:01,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-24 13:58:01,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-24 13:58:01,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-24 13:58:01,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-24 13:58:01,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-24 13:58:01,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-24 13:58:01,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-24 13:58:01,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-24 13:58:01,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-24 13:58:01,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-24 13:58:01,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-24 13:58:01,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-24 13:58:01,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-24 13:58:01,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-24 13:58:01,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-24 13:58:01,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-24 13:58:01,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-24 13:58:01,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-24 13:58:01,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-24 13:58:01,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-24 13:58:01,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-24 13:58:01,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-24 13:58:01,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-24 13:58:01,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-24 13:58:01,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-24 13:58:01,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-24 13:58:01,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-24 13:58:01,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-24 13:58:01,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-24 13:58:01,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-24 13:58:01,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-24 13:58:01,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-24 13:58:01,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-24 13:58:01,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-24 13:58:01,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-24 13:58:01,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-24 13:58:01,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-24 13:58:01,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-24 13:58:01,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-24 13:58:01,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-24 13:58:01,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-24 13:58:01,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-24 13:58:01,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-24 13:58:01,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-24 13:58:01,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-24 13:58:01,394 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-24 13:58:01,394 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-24 13:58:01,394 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-24 13:58:01,823 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 622 ms to try 5000 different iterations. Found a solution that moves 4682 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17870431621700753. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038928471117859374, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05594208966586843, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5202222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038928471117859374, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17391666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24058333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-24 13:58:01,856 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv115543251:81 , srv2064444310:82 , srv582377435:82 , srv106941380:83 , srv1615643684:83 , srv1899698486:83 , srv2034479452:83 , srv316056927:83 , srv384136672:83 , srv443996363:83 , srv550978423:83 , srv559438311:83 , srv1076099564:84 , srv1328827719:84 , srv1382031616:84 , srv1934731585:84 , srv380614764:84 , srv721999540:84 , srv969586164:84 , srv1089509270:85 , srv1299000668:85 , srv1327636093:85 , srv1434237857:85 , srv1768123917:85 , srv1833649697:85 , srv1949212252:85 , srv232499113:85 , srv316870931:85 , srv512073002:85 , srv694347450:85 , srv872281481:85 , srv897078711:85 , srv95033233:85 , srv995965519:85 , srv1001604145:86 , srv1139509108:86 , srv1157760957:86 , srv1274399778:86 , srv1367992234:86 , srv1405332683:86 , srv145341218:86 , srv1463792196:86 , srv1748596458:86 , srv180278264:86 , srv1999459271:86 , srv2075193292:86 , srv237441112:86 , srv336628879:86 , srv709584708:86 , srv855219536:86 , srv967388585:86 , srv1004016818:87 , srv1018285316:87 , srv1178261019:87 , srv1246818358:87 , srv1291973037:87 , srv1407730602:87 , srv1451012082:87 , srv1544006684:87 , srv1583419286:87 , srv1691994868:87 , srv1873148042:87 , srv2036324752:87 , srv2048038018:87 , srv209550640:87 , srv554673701:87 , srv657777772:87 , srv741579659:87 , srv771962791:87 , srv915561945:87 , srv123135318:88 , srv137615500:88 , srv1754711791:88 , srv1827213846:88 , srv1876257834:88 , srv2020782619:88 , srv2028630977:88 , srv208702260:88 , srv646980193:88 , srv134364819:89 , srv1918082644:89 , srv491637507:89 , srv615756302:89 , srv751483481:89 , srv791563248:89 , srv979420311:89 , srv99696301:89 , srv1212590300:90 , srv1284185606:90 , srv1297992494:90 , srv1318634086:90 , srv1330788157:90 , srv1364436596:90 , srv227485484:90 , srv757263530:90 , srv1703334845:91 , srv1744179849:91 , srv589713490:92 , srv1522232073:96 , srv103458101:436 } 2023-08-24 13:58:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-24 13:58:01,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1076099564=5, srv1949212252=52, srv2036324752=57, srv550978423=74, srv95033233=94, srv872281481=91, srv915561945=93, srv1297992494=17, srv721999540=84, srv1299000668=18, srv1405332683=28, srv134364819=23, srv709584708=83, srv1463792196=33, srv99696301=99, srv2034479452=56, srv855219536=90, srv657777772=81, srv1212590300=11, srv967388585=95, srv1918082644=50, srv559438311=76, srv1522232073=34, srv646980193=80, srv615756302=79, srv443996363=71, srv1768123917=43, srv145341218=32, srv1407730602=29, srv1364436596=24, srv1833649697=46, srv106941380=4, srv1691994868=38, srv336628879=68, srv1544006684=35, srv115543251=8, srv316870931=67, srv1318634086=19, srv1274399778=14, srv1615643684=37, srv1873148042=47, srv1291973037=16, srv757263530=87, srv554673701=75, srv1876257834=48, srv180278264=44, srv384136672=70, srv1382031616=27, srv995965519=98, srv1001604145=0, srv1754711791=42, srv208702260=61, srv491637507=72, srv1744179849=40, srv137615500=26, srv1899698486=49, srv1748596458=41, srv1330788157=22, srv751483481=86, srv2064444310=59, srv741579659=85, srv791563248=89, srv1451012082=31, srv979420311=97, srv1367992234=25, srv969586164=96, srv1089509270=6, srv237441112=65, srv1139509108=7, srv589713490=78, srv1703334845=39, srv1583419286=36, srv380614764=69, srv1004016818=1, srv2020782619=54, srv1827213846=45, srv897078711=92, srv694347450=82, srv2075193292=60, srv227485484=63, srv1246818358=13, srv1157760957=9, srv209550640=62, srv771962791=88, srv1434237857=30, srv2028630977=55, srv232499113=64, srv2048038018=58, srv1999459271=53, srv512073002=73, srv1178261019=10, srv1934731585=51, srv582377435=77, srv1328827719=21, srv316056927=66, srv103458101=3, srv123135318=12, srv1018285316=2, srv1284185606=15, srv1327636093=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-24 13:58:01,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-24 13:58:01,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-24 13:58:01,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-24 13:58:01,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-24 13:58:01,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-24 13:58:01,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-24 13:58:01,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-24 13:58:01,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-24 13:58:01,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-24 13:58:01,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-24 13:58:01,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-24 13:58:01,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-24 13:58:01,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-24 13:58:01,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-24 13:58:01,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-24 13:58:01,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-24 13:58:01,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-24 13:58:01,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-24 13:58:01,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-24 13:58:01,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-24 13:58:01,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-24 13:58:01,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-24 13:58:01,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-24 13:58:01,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-24 13:58:01,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-24 13:58:01,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-24 13:58:01,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-24 13:58:01,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-24 13:58:01,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-24 13:58:01,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-24 13:58:01,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-24 13:58:01,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-24 13:58:01,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-24 13:58:01,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-24 13:58:01,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-24 13:58:01,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-24 13:58:01,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-24 13:58:01,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-24 13:58:01,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-24 13:58:01,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-24 13:58:01,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-24 13:58:01,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-24 13:58:01,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-24 13:58:01,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-24 13:58:01,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-24 13:58:01,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-24 13:58:01,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-24 13:58:01,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-24 13:58:01,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-24 13:58:01,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-24 13:58:01,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-24 13:58:01,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-24 13:58:01,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-24 13:58:01,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-24 13:58:01,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-24 13:58:01,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-24 13:58:01,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-24 13:58:01,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-24 13:58:01,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-24 13:58:01,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-24 13:58:01,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-24 13:58:01,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-24 13:58:01,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-24 13:58:01,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-24 13:58:01,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-24 13:58:01,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-24 13:58:01,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-24 13:58:01,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-24 13:58:01,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-24 13:58:01,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-24 13:58:01,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-24 13:58:01,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-24 13:58:01,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-24 13:58:01,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-24 13:58:01,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-24 13:58:01,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-24 13:58:01,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-24 13:58:01,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-24 13:58:01,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-24 13:58:01,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-24 13:58:01,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-24 13:58:01,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-24 13:58:01,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-24 13:58:01,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-24 13:58:01,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-24 13:58:01,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-24 13:58:01,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-24 13:58:01,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-24 13:58:01,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-24 13:58:01,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-24 13:58:01,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-24 13:58:01,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-24 13:58:01,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-24 13:58:01,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-24 13:58:01,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-24 13:58:01,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-24 13:58:01,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-24 13:58:01,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-24 13:58:01,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-24 13:58:01,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-24 13:58:01,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-24 13:58:02,001 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038928471117859374, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05594208966586843, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038928471117859374, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17391666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24058333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-24 13:58:02,002 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-24 13:58:02,002 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17867153215229967, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038928471117859374, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05594208966586843, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038928471117859374, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17391666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24058333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-24 13:58:02,358 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 460 ms to try 5000 different iterations. Found a solution that moves 2596 regions; Going from a computed imbalance of 0.17867153215229967 to a new imbalance of 0.015960783084897474. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021159787936423258); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03407261012811846, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28844444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021159787936423258); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-24 13:58:02,373 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1522232073:77 , srv589713490:78 , srv1318634086:81 , srv491637507:81 , srv751483481:81 , srv757263530:81 , srv1691994868:82 , srv1744179849:82 , srv1827213846:82 , srv2020782619:82 , srv615756302:82 , srv1157760957:83 , srv1212590300:83 , srv1284185606:83 , srv1297992494:83 , srv1583419286:83 , srv1703334845:83 , srv1918082644:83 , srv227485484:83 , srv646980193:83 , srv979420311:83 , srv1004016818:84 , srv1018285316:84 , srv123135318:84 , srv1330788157:84 , srv134364819:84 , srv1434237857:84 , srv1451012082:84 , srv1754711791:84 , srv208702260:84 , srv554673701:84 , srv791563248:84 , srv1178261019:85 , srv1274399778:85 , srv1291973037:85 , srv1873148042:85 , srv1999459271:85 , srv2028630977:85 , srv2036324752:85 , srv209550640:85 , srv237441112:85 , srv336628879:85 , srv915561945:85 , srv99696301:85 , srv106941380:86 , srv1139509108:86 , srv1299000668:86 , srv1364436596:86 , srv1367992234:86 , srv137615500:86 , srv145341218:86 , srv1463792196:86 , srv1544006684:86 , srv1768123917:86 , srv1876257834:86 , srv1934731585:86 , srv2048038018:86 , srv316870931:86 , srv741579659:86 , srv855219536:86 , srv967388585:86 , srv1001604145:87 , srv1076099564:87 , srv1089509270:87 , srv1246818358:87 , srv1327636093:87 , srv1328827719:87 , srv1382031616:87 , srv1407730602:87 , srv657777772:87 , srv694347450:87 , srv771962791:87 , srv872281481:87 , srv969586164:87 , srv1748596458:88 , srv180278264:88 , srv2034479452:88 , srv2075193292:88 , srv232499113:88 , srv380614764:88 , srv443996363:88 , srv512073002:88 , srv550978423:88 , srv709584708:88 , srv897078711:88 , srv95033233:88 , srv1405332683:89 , srv1833649697:89 , srv1899698486:89 , srv2064444310:89 , srv316056927:89 , srv384136672:89 , srv559438311:89 , srv995965519:89 , srv115543251:90 , srv1615643684:90 , srv1949212252:90 , srv582377435:90 , srv721999540:90 , srv103458101:526 } 2023-08-24 13:58:02,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-24 13:58:02,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1076099564=5, srv1949212252=52, srv2036324752=57, srv550978423=74, srv95033233=94, srv872281481=91, srv915561945=93, srv1297992494=17, srv721999540=84, srv1299000668=18, srv1405332683=28, srv134364819=23, srv709584708=83, srv1463792196=33, srv99696301=99, srv2034479452=56, srv855219536=90, srv657777772=81, srv1212590300=11, srv967388585=95, srv1918082644=50, srv559438311=76, srv1522232073=34, srv646980193=80, srv615756302=79, srv443996363=71, srv1768123917=43, srv145341218=32, srv1407730602=29, srv1364436596=24, srv1833649697=46, srv106941380=4, srv1691994868=38, srv336628879=68, srv1544006684=35, srv115543251=8, srv316870931=67, srv1318634086=19, srv1274399778=14, srv1615643684=37, srv1873148042=47, srv1291973037=16, srv757263530=87, srv554673701=75, srv1876257834=48, srv180278264=44, srv384136672=70, srv1382031616=27, srv995965519=98, srv1001604145=0, srv1754711791=42, srv208702260=61, srv491637507=72, srv1744179849=40, srv137615500=26, srv1899698486=49, srv1748596458=41, srv1330788157=22, srv751483481=86, srv2064444310=59, srv741579659=85, srv791563248=89, srv1451012082=31, srv979420311=97, srv1367992234=25, srv969586164=96, srv1089509270=6, srv237441112=65, srv1139509108=7, srv589713490=78, srv1703334845=39, srv1583419286=36, srv380614764=69, srv1004016818=1, srv2020782619=54, srv1827213846=45, srv897078711=92, srv694347450=82, srv2075193292=60, srv227485484=63, srv1246818358=13, srv1157760957=9, srv209550640=62, srv771962791=88, srv1434237857=30, srv2028630977=55, srv232499113=64, srv2048038018=58, srv1999459271=53, srv512073002=73, srv1178261019=10, srv1934731585=51, srv582377435=77, srv1328827719=21, srv316056927=66, srv103458101=3, srv123135318=12, srv1018285316=2, srv1284185606=15, srv1327636093=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-24 13:58:02,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-24 13:58:02,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-24 13:58:02,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-24 13:58:02,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-24 13:58:02,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-24 13:58:02,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-24 13:58:02,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-24 13:58:02,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-24 13:58:02,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-24 13:58:02,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-24 13:58:02,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-24 13:58:02,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-24 13:58:02,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-24 13:58:02,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-24 13:58:02,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-24 13:58:02,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-24 13:58:02,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-24 13:58:02,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-24 13:58:02,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-24 13:58:02,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-24 13:58:02,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-24 13:58:02,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-24 13:58:02,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-24 13:58:02,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-24 13:58:02,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-24 13:58:02,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-24 13:58:02,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-24 13:58:02,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-24 13:58:02,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-24 13:58:02,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-24 13:58:02,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-24 13:58:02,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-24 13:58:02,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-24 13:58:02,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-24 13:58:02,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-24 13:58:02,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-24 13:58:02,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-24 13:58:02,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-24 13:58:02,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-24 13:58:02,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-24 13:58:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-24 13:58:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-24 13:58:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-24 13:58:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-24 13:58:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-24 13:58:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-24 13:58:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-24 13:58:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-24 13:58:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-24 13:58:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-24 13:58:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-24 13:58:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-24 13:58:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-24 13:58:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-24 13:58:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-24 13:58:02,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-24 13:58:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-24 13:58:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-24 13:58:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-24 13:58:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-24 13:58:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-24 13:58:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-24 13:58:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-24 13:58:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-24 13:58:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-24 13:58:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-24 13:58:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-24 13:58:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-24 13:58:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-24 13:58:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-24 13:58:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-24 13:58:02,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-24 13:58:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-24 13:58:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-24 13:58:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-24 13:58:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-24 13:58:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-24 13:58:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-24 13:58:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-24 13:58:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-24 13:58:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-24 13:58:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-24 13:58:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-24 13:58:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-24 13:58:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-24 13:58:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-24 13:58:02,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-24 13:58:02,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-24 13:58:02,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-24 13:58:02,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-24 13:58:02,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-24 13:58:02,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-24 13:58:02,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-24 13:58:02,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-24 13:58:02,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-24 13:58:02,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-24 13:58:02,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-24 13:58:02,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-24 13:58:02,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-24 13:58:02,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-24 13:58:02,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-24 13:58:02,578 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021159787936423258); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03407261012811846, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021159787936423258); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-24 13:58:02,578 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-24 13:58:02,578 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01594260550438026, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021159787936423258); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03407261012811846, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021159787936423258); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-24 13:58:02,882 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 431 ms to try 5000 different iterations. Found a solution that moves 1011 regions; Going from a computed imbalance of 0.01594260550438026 to a new imbalance of 7.393976760311027E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006444096124338012); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009125635134638571); MoveCostFunction : (multiplier=7.0, imbalance=0.11233333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006444096124338012); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-24 13:58:02,884 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1364436596:81 , srv1405332683:82 , srv1703334845:82 , srv1246818358:83 , srv1330788157:83 , srv137615500:83 , srv1748596458:83 , srv180278264:83 , srv1833649697:83 , srv1949212252:83 , srv2075193292:83 , srv227485484:83 , srv232499113:83 , srv721999540:83 , srv95033233:83 , srv995965519:83 , srv1297992494:84 , srv1407730602:84 , srv512073002:84 , srv657777772:84 , srv741579659:84 , srv771962791:84 , srv791563248:84 , srv99696301:84 , srv1001604145:85 , srv1139509108:85 , srv1212590300:85 , srv1284185606:85 , srv1291973037:85 , srv1318634086:85 , srv134364819:85 , srv1367992234:85 , srv1522232073:85 , srv1544006684:85 , srv1615643684:85 , srv1754711791:85 , srv1876257834:85 , srv1899698486:85 , srv1918082644:85 , srv2028630977:85 , srv2048038018:85 , srv209550640:85 , srv316056927:85 , srv559438311:85 , srv582377435:85 , srv694347450:85 , srv709584708:85 , srv855219536:85 , srv897078711:85 , srv915561945:85 , srv979420311:85 , srv1004016818:86 , srv1018285316:86 , srv1076099564:86 , srv1089509270:86 , srv1178261019:86 , srv123135318:86 , srv1274399778:86 , srv1451012082:86 , srv145341218:86 , srv1463792196:86 , srv1744179849:86 , srv1873148042:86 , srv2036324752:86 , srv2064444310:86 , srv208702260:86 , srv237441112:86 , srv316870931:86 , srv336628879:86 , srv380614764:86 , srv384136672:86 , srv443996363:86 , srv554673701:86 , srv615756302:86 , srv646980193:86 , srv872281481:86 , srv967388585:86 , srv969586164:86 , srv115543251:87 , srv1299000668:87 , srv1327636093:87 , srv1328827719:87 , srv1382031616:87 , srv1691994868:87 , srv1827213846:87 , srv1934731585:87 , srv1999459271:87 , srv2020782619:87 , srv2034479452:87 , srv491637507:87 , srv550978423:87 , srv589713490:87 , srv751483481:87 , srv1157760957:88 , srv1434237857:88 , srv1583419286:88 , srv1768123917:88 , srv757263530:88 , srv106941380:89 , srv103458101:553 } 2023-08-24 13:58:02,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-24 13:58:02,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1076099564=5, srv1949212252=52, srv2036324752=57, srv550978423=74, srv95033233=94, srv872281481=91, srv915561945=93, srv1297992494=17, srv721999540=84, srv1299000668=18, srv1405332683=28, srv134364819=23, srv709584708=83, srv1463792196=33, srv99696301=99, srv2034479452=56, srv855219536=90, srv657777772=81, srv1212590300=11, srv967388585=95, srv1918082644=50, srv559438311=76, srv1522232073=34, srv646980193=80, srv615756302=79, srv443996363=71, srv1768123917=43, srv145341218=32, srv1407730602=29, srv1364436596=24, srv1833649697=46, srv106941380=4, srv1691994868=38, srv336628879=68, srv1544006684=35, srv115543251=8, srv316870931=67, srv1318634086=19, srv1274399778=14, srv1615643684=37, srv1873148042=47, srv1291973037=16, srv757263530=87, srv554673701=75, srv1876257834=48, srv180278264=44, srv384136672=70, srv1382031616=27, srv995965519=98, srv1001604145=0, srv1754711791=42, srv208702260=61, srv491637507=72, srv1744179849=40, srv137615500=26, srv1899698486=49, srv1748596458=41, srv1330788157=22, srv751483481=86, srv2064444310=59, srv741579659=85, srv791563248=89, srv1451012082=31, srv979420311=97, srv1367992234=25, srv969586164=96, srv1089509270=6, srv237441112=65, srv1139509108=7, srv589713490=78, srv1703334845=39, srv1583419286=36, srv380614764=69, srv1004016818=1, srv2020782619=54, srv1827213846=45, srv897078711=92, srv694347450=82, srv2075193292=60, srv227485484=63, srv1246818358=13, srv1157760957=9, srv209550640=62, srv771962791=88, srv1434237857=30, srv2028630977=55, srv232499113=64, srv2048038018=58, srv1999459271=53, srv512073002=73, srv1178261019=10, srv1934731585=51, srv582377435=77, srv1328827719=21, srv316056927=66, srv103458101=3, srv123135318=12, srv1018285316=2, srv1284185606=15, srv1327636093=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-24 13:58:02,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-24 13:58:02,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-24 13:58:02,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-24 13:58:02,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-24 13:58:02,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-24 13:58:02,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-24 13:58:02,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-24 13:58:02,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-24 13:58:02,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-24 13:58:02,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-24 13:58:02,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-24 13:58:02,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-24 13:58:02,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-24 13:58:02,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-24 13:58:02,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-24 13:58:02,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-24 13:58:02,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-24 13:58:02,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-24 13:58:02,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-24 13:58:02,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-24 13:58:02,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-24 13:58:02,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-24 13:58:02,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-24 13:58:02,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-24 13:58:02,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-24 13:58:02,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-24 13:58:02,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-24 13:58:02,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-24 13:58:02,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-24 13:58:02,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-24 13:58:02,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-24 13:58:02,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-24 13:58:02,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-24 13:58:02,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-24 13:58:02,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-24 13:58:02,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-24 13:58:02,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-24 13:58:02,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-24 13:58:02,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-24 13:58:02,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-24 13:58:02,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-24 13:58:02,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-24 13:58:02,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-24 13:58:02,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-24 13:58:02,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-24 13:58:02,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-24 13:58:02,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-24 13:58:02,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-24 13:58:02,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-24 13:58:02,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-24 13:58:02,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-24 13:58:02,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-24 13:58:02,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-24 13:58:02,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-24 13:58:02,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-24 13:58:02,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-24 13:58:02,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-24 13:58:02,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-24 13:58:02,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-24 13:58:02,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-24 13:58:02,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-24 13:58:02,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-24 13:58:02,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-24 13:58:02,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-24 13:58:02,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-24 13:58:02,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-24 13:58:02,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-24 13:58:02,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-24 13:58:02,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-24 13:58:02,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-24 13:58:02,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-24 13:58:02,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-24 13:58:02,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-24 13:58:02,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-24 13:58:02,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-24 13:58:02,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-24 13:58:02,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-24 13:58:02,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-24 13:58:02,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-24 13:58:02,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-24 13:58:02,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-24 13:58:02,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-24 13:58:02,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-24 13:58:02,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-24 13:58:02,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-24 13:58:02,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-24 13:58:02,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-24 13:58:02,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-24 13:58:02,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-24 13:58:02,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-24 13:58:02,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-24 13:58:02,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-24 13:58:02,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-24 13:58:02,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-24 13:58:02,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-24 13:58:02,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-24 13:58:02,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-24 13:58:02,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-24 13:58:02,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-24 13:58:02,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-24 13:58:02,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-24 13:58:03,019 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006444096124338012); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009125635134638571); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006444096124338012); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-24 13:58:03,019 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-24 13:58:03,019 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.323185027248977E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006444096124338012); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009125635134638571); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006444096124338012); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-24 13:58:03,462 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 543 ms to try 5000 different iterations. Found a solution that moves 254 regions; Going from a computed imbalance of 7.323185027248977E-4 to a new imbalance of 2.5993873043552464E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.02822222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-24 13:58:03,463 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1299000668:83 , srv1327636093:83 , srv1544006684:83 , srv1744179849:83 , srv1768123917:83 , srv1876257834:83 , srv709584708:83 , srv757263530:83 , srv1001604145:84 , srv106941380:84 , srv1076099564:84 , srv1089509270:84 , srv115543251:84 , srv1178261019:84 , srv1212590300:84 , srv123135318:84 , srv1284185606:84 , srv1328827719:84 , srv134364819:84 , srv1382031616:84 , srv1407730602:84 , srv145341218:84 , srv1463792196:84 , srv1522232073:84 , srv1615643684:84 , srv1873148042:84 , srv1949212252:84 , srv1999459271:84 , srv2028630977:84 , srv2034479452:84 , srv2036324752:84 , srv2048038018:84 , srv208702260:84 , srv380614764:84 , srv384136672:84 , srv550978423:84 , srv554673701:84 , srv872281481:84 , srv897078711:84 , srv967388585:84 , srv99696301:84 , srv1004016818:85 , srv1018285316:85 , srv1139509108:85 , srv1157760957:85 , srv1246818358:85 , srv1274399778:85 , srv1291973037:85 , srv1297992494:85 , srv1330788157:85 , srv1364436596:85 , srv1367992234:85 , srv137615500:85 , srv1405332683:85 , srv1434237857:85 , srv1451012082:85 , srv1583419286:85 , srv1748596458:85 , srv1754711791:85 , srv180278264:85 , srv1827213846:85 , srv1833649697:85 , srv1899698486:85 , srv1918082644:85 , srv1934731585:85 , srv2020782619:85 , srv2064444310:85 , srv2075193292:85 , srv209550640:85 , srv237441112:85 , srv316056927:85 , srv316870931:85 , srv336628879:85 , srv443996363:85 , srv491637507:85 , srv512073002:85 , srv559438311:85 , srv582377435:85 , srv589713490:85 , srv615756302:85 , srv646980193:85 , srv657777772:85 , srv694347450:85 , srv741579659:85 , srv751483481:85 , srv771962791:85 , srv791563248:85 , srv855219536:85 , srv915561945:85 , srv969586164:85 , srv979420311:85 , srv995965519:85 , srv1318634086:86 , srv1691994868:86 , srv1703334845:86 , srv227485484:86 , srv232499113:86 , srv721999540:86 , srv95033233:86 , srv103458101:627 } 2023-08-24 13:58:03,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-24 13:58:03,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1076099564=5, srv1949212252=52, srv2036324752=57, srv550978423=74, srv95033233=94, srv872281481=91, srv915561945=93, srv1297992494=17, srv721999540=84, srv1299000668=18, srv1405332683=28, srv134364819=23, srv709584708=83, srv1463792196=33, srv99696301=99, srv2034479452=56, srv855219536=90, srv657777772=81, srv1212590300=11, srv967388585=95, srv1918082644=50, srv559438311=76, srv1522232073=34, srv646980193=80, srv615756302=79, srv443996363=71, srv1768123917=43, srv145341218=32, srv1407730602=29, srv1364436596=24, srv1833649697=46, srv106941380=4, srv1691994868=38, srv336628879=68, srv1544006684=35, srv115543251=8, srv316870931=67, srv1318634086=19, srv1274399778=14, srv1615643684=37, srv1873148042=47, srv1291973037=16, srv757263530=87, srv554673701=75, srv1876257834=48, srv180278264=44, srv384136672=70, srv1382031616=27, srv995965519=98, srv1001604145=0, srv1754711791=42, srv208702260=61, srv491637507=72, srv1744179849=40, srv137615500=26, srv1899698486=49, srv1748596458=41, srv1330788157=22, srv751483481=86, srv2064444310=59, srv741579659=85, srv791563248=89, srv1451012082=31, srv979420311=97, srv1367992234=25, srv969586164=96, srv1089509270=6, srv237441112=65, srv1139509108=7, srv589713490=78, srv1703334845=39, srv1583419286=36, srv380614764=69, srv1004016818=1, srv2020782619=54, srv1827213846=45, srv897078711=92, srv694347450=82, srv2075193292=60, srv227485484=63, srv1246818358=13, srv1157760957=9, srv209550640=62, srv771962791=88, srv1434237857=30, srv2028630977=55, srv232499113=64, srv2048038018=58, srv1999459271=53, srv512073002=73, srv1178261019=10, srv1934731585=51, srv582377435=77, srv1328827719=21, srv316056927=66, srv103458101=3, srv123135318=12, srv1018285316=2, srv1284185606=15, srv1327636093=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-24 13:58:03,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-24 13:58:03,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-24 13:58:03,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-24 13:58:03,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-24 13:58:03,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-24 13:58:03,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-24 13:58:03,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-24 13:58:03,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-24 13:58:03,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-24 13:58:03,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-24 13:58:03,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-24 13:58:03,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-24 13:58:03,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-24 13:58:03,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-24 13:58:03,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-24 13:58:03,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-24 13:58:03,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-24 13:58:03,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-24 13:58:03,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-24 13:58:03,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-24 13:58:03,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-24 13:58:03,489 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-24 13:58:03,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-24 13:58:03,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-24 13:58:03,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-24 13:58:03,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-24 13:58:03,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-24 13:58:03,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-24 13:58:03,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-24 13:58:03,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-24 13:58:03,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-24 13:58:03,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-24 13:58:03,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-24 13:58:03,490 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-24 13:58:03,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-24 13:58:03,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-24 13:58:03,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-24 13:58:03,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-24 13:58:03,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-24 13:58:03,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-24 13:58:03,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-24 13:58:03,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-24 13:58:03,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-24 13:58:03,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-24 13:58:03,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-24 13:58:03,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-24 13:58:03,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-24 13:58:03,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-24 13:58:03,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-24 13:58:03,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-24 13:58:03,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-24 13:58:03,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-24 13:58:03,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-24 13:58:03,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-24 13:58:03,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-24 13:58:03,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-24 13:58:03,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-24 13:58:03,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-24 13:58:03,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-24 13:58:03,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-24 13:58:03,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-24 13:58:03,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-24 13:58:03,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-24 13:58:03,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-24 13:58:03,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-24 13:58:03,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-24 13:58:03,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-24 13:58:03,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-24 13:58:03,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-24 13:58:03,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-24 13:58:03,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-24 13:58:03,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-24 13:58:03,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-24 13:58:03,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-24 13:58:03,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-24 13:58:03,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-24 13:58:03,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-24 13:58:03,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-24 13:58:03,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-24 13:58:03,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-24 13:58:03,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-24 13:58:03,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-24 13:58:03,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-24 13:58:03,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-24 13:58:03,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-24 13:58:03,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-24 13:58:03,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-24 13:58:03,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-24 13:58:03,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-24 13:58:03,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-24 13:58:03,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-24 13:58:03,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-24 13:58:03,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-24 13:58:03,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-24 13:58:03,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-24 13:58:03,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-24 13:58:03,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-24 13:58:03,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-24 13:58:03,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-24 13:58:03,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-24 13:58:03,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-24 13:58:03,653 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-24 13:58:03,653 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-24 13:58:03,653 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.421532702992628E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-24 13:58:03,964 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 455 ms to try 5000 different iterations. Found a solution that moves 12 regions; Going from a computed imbalance of 2.421532702992628E-5 to a new imbalance of 7.4717832962319055E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0013333333333333333); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-24 13:58:03,965 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1583419286:83 , srv657777772:83 , srv721999540:83 , srv771962791:83 , srv1001604145:84 , srv1004016818:84 , srv1018285316:84 , srv106941380:84 , srv1089509270:84 , srv1139509108:84 , srv115543251:84 , srv1157760957:84 , srv1178261019:84 , srv1212590300:84 , srv123135318:84 , srv1246818358:84 , srv1274399778:84 , srv1284185606:84 , srv1291973037:84 , srv1297992494:84 , srv1318634086:84 , srv1327636093:84 , srv1328827719:84 , srv1330788157:84 , srv134364819:84 , srv1364436596:84 , srv1367992234:84 , srv137615500:84 , srv1382031616:84 , srv1405332683:84 , srv1407730602:84 , srv1434237857:84 , srv1451012082:84 , srv145341218:84 , srv1463792196:84 , srv1522232073:84 , srv1615643684:84 , srv1691994868:84 , srv1703334845:84 , srv1744179849:84 , srv1748596458:84 , srv1754711791:84 , srv1768123917:84 , srv180278264:84 , srv1827213846:84 , srv1833649697:84 , srv1873148042:84 , srv1876257834:84 , srv1899698486:84 , srv1918082644:84 , srv1934731585:84 , srv1949212252:84 , srv1999459271:84 , srv2020782619:84 , srv2028630977:84 , srv2034479452:84 , srv2036324752:84 , srv2048038018:84 , srv2064444310:84 , srv2075193292:84 , srv208702260:84 , srv209550640:84 , srv227485484:84 , srv232499113:84 , srv237441112:84 , srv316056927:84 , srv316870931:84 , srv336628879:84 , srv380614764:84 , srv384136672:84 , srv443996363:84 , srv491637507:84 , srv512073002:84 , srv550978423:84 , srv559438311:84 , srv582377435:84 , srv589713490:84 , srv615756302:84 , srv646980193:84 , srv694347450:84 , srv709584708:84 , srv741579659:84 , srv751483481:84 , srv757263530:84 , srv791563248:84 , srv855219536:84 , srv872281481:84 , srv897078711:84 , srv915561945:84 , srv95033233:84 , srv967388585:84 , srv969586164:84 , srv979420311:84 , srv995965519:84 , srv99696301:84 , srv1076099564:85 , srv1299000668:85 , srv1544006684:85 , srv554673701:85 , srv103458101:684 } 2023-08-24 13:58:03,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-24 13:58:03,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1076099564=5, srv1949212252=52, srv2036324752=57, srv550978423=74, srv95033233=94, srv872281481=91, srv915561945=93, srv1297992494=17, srv721999540=84, srv1299000668=18, srv1405332683=28, srv134364819=23, srv709584708=83, srv1463792196=33, srv99696301=99, srv2034479452=56, srv855219536=90, srv657777772=81, srv1212590300=11, srv967388585=95, srv1918082644=50, srv559438311=76, srv1522232073=34, srv646980193=80, srv615756302=79, srv443996363=71, srv1768123917=43, srv145341218=32, srv1407730602=29, srv1364436596=24, srv1833649697=46, srv106941380=4, srv1691994868=38, srv336628879=68, srv1544006684=35, srv115543251=8, srv316870931=67, srv1318634086=19, srv1274399778=14, srv1615643684=37, srv1873148042=47, srv1291973037=16, srv757263530=87, srv554673701=75, srv1876257834=48, srv180278264=44, srv384136672=70, srv1382031616=27, srv995965519=98, srv1001604145=0, srv1754711791=42, srv208702260=61, srv491637507=72, srv1744179849=40, srv137615500=26, srv1899698486=49, srv1748596458=41, srv1330788157=22, srv751483481=86, srv2064444310=59, srv741579659=85, srv791563248=89, srv1451012082=31, srv979420311=97, srv1367992234=25, srv969586164=96, srv1089509270=6, srv237441112=65, srv1139509108=7, srv589713490=78, srv1703334845=39, srv1583419286=36, srv380614764=69, srv1004016818=1, srv2020782619=54, srv1827213846=45, srv897078711=92, srv694347450=82, srv2075193292=60, srv227485484=63, srv1246818358=13, srv1157760957=9, srv209550640=62, srv771962791=88, srv1434237857=30, srv2028630977=55, srv232499113=64, srv2048038018=58, srv1999459271=53, srv512073002=73, srv1178261019=10, srv1934731585=51, srv582377435=77, srv1328827719=21, srv316056927=66, srv103458101=3, srv123135318=12, srv1018285316=2, srv1284185606=15, srv1327636093=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-24 13:58:03,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-24 13:58:03,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-24 13:58:03,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-24 13:58:03,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-24 13:58:03,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-24 13:58:03,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-24 13:58:03,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-24 13:58:03,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-24 13:58:03,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-24 13:58:03,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-24 13:58:03,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-24 13:58:03,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-24 13:58:03,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-24 13:58:03,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-24 13:58:03,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-24 13:58:03,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-24 13:58:03,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-24 13:58:03,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-24 13:58:03,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-24 13:58:03,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-24 13:58:03,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-24 13:58:03,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-24 13:58:03,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-24 13:58:03,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-24 13:58:03,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-24 13:58:03,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-24 13:58:03,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-24 13:58:03,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-24 13:58:03,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-24 13:58:03,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-24 13:58:03,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-24 13:58:03,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-24 13:58:03,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-24 13:58:03,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-24 13:58:03,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-24 13:58:03,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-24 13:58:03,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-24 13:58:03,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-24 13:58:03,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-24 13:58:03,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-24 13:58:03,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-24 13:58:03,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-24 13:58:03,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-24 13:58:03,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-24 13:58:03,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-24 13:58:03,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-24 13:58:03,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-24 13:58:03,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-24 13:58:03,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-24 13:58:03,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-24 13:58:03,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-24 13:58:03,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-24 13:58:03,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-24 13:58:03,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-24 13:58:03,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-24 13:58:03,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-24 13:58:03,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-24 13:58:03,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-24 13:58:03,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-24 13:58:03,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-24 13:58:03,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-24 13:58:03,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-24 13:58:03,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-24 13:58:03,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-24 13:58:03,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-24 13:58:03,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-24 13:58:03,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-24 13:58:03,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-24 13:58:03,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-24 13:58:03,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-24 13:58:03,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-24 13:58:03,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-24 13:58:03,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-24 13:58:03,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-24 13:58:03,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-24 13:58:03,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-24 13:58:03,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-24 13:58:03,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-24 13:58:03,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-24 13:58:03,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-24 13:58:03,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-24 13:58:03,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-24 13:58:03,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-24 13:58:03,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-24 13:58:03,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-24 13:58:03,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-24 13:58:03,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-24 13:58:03,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-24 13:58:03,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-24 13:58:03,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-24 13:58:03,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-24 13:58:03,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-24 13:58:03,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-24 13:58:03,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-24 13:58:03,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-24 13:58:03,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-24 13:58:03,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-24 13:58:03,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-24 13:58:03,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-24 13:58:03,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-24 13:58:03,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-24 13:58:04,144 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-24 13:58:04,145 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1583419286:83 , srv657777772:83 , srv721999540:83 , srv771962791:83 , srv1001604145:84 , srv1004016818:84 , srv1018285316:84 , srv106941380:84 , srv1089509270:84 , srv1139509108:84 , srv115543251:84 , srv1157760957:84 , srv1178261019:84 , srv1212590300:84 , srv123135318:84 , srv1246818358:84 , srv1274399778:84 , srv1284185606:84 , srv1291973037:84 , srv1297992494:84 , srv1318634086:84 , srv1327636093:84 , srv1328827719:84 , srv1330788157:84 , srv134364819:84 , srv1364436596:84 , srv1367992234:84 , srv137615500:84 , srv1382031616:84 , srv1405332683:84 , srv1407730602:84 , srv1434237857:84 , srv1451012082:84 , srv145341218:84 , srv1463792196:84 , srv1522232073:84 , srv1615643684:84 , srv1691994868:84 , srv1703334845:84 , srv1744179849:84 , srv1748596458:84 , srv1754711791:84 , srv1768123917:84 , srv180278264:84 , srv1827213846:84 , srv1833649697:84 , srv1873148042:84 , srv1876257834:84 , srv1899698486:84 , srv1918082644:84 , srv1934731585:84 , srv1949212252:84 , srv1999459271:84 , srv2020782619:84 , srv2028630977:84 , srv2034479452:84 , srv2036324752:84 , srv2048038018:84 , srv2064444310:84 , srv2075193292:84 , srv208702260:84 , srv209550640:84 , srv227485484:84 , srv232499113:84 , srv237441112:84 , srv316056927:84 , srv316870931:84 , srv336628879:84 , srv380614764:84 , srv384136672:84 , srv443996363:84 , srv491637507:84 , srv512073002:84 , srv550978423:84 , srv559438311:84 , srv582377435:84 , srv589713490:84 , srv615756302:84 , srv646980193:84 , srv694347450:84 , srv709584708:84 , srv741579659:84 , srv751483481:84 , srv757263530:84 , srv791563248:84 , srv855219536:84 , srv872281481:84 , srv897078711:84 , srv915561945:84 , srv95033233:84 , srv967388585:84 , srv969586164:84 , srv979420311:84 , srv995965519:84 , srv99696301:84 , srv1076099564:85 , srv1299000668:85 , srv1544006684:85 , srv554673701:85 , srv103458101:684 } 2023-08-24 13:58:04,181 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=327 (was 303) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=10723 (was 11582) 2023-08-24 13:58:04,191 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=327, ProcessCount=172, AvailableMemoryMB=10722 2023-08-24 13:58:04,192 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-24 13:58:04,192 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-08-24 13:58:04,218 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1013795781:84 , srv1159796009:84 , srv149360983:84 , srv1542331641:84 , srv168433757:84 , srv1701149567:84 , srv1744780700:84 , srv1778914283:84 , srv1844091921:84 , srv1846002324:84 , srv1854703096:84 , srv1860933127:84 , srv190245973:84 , srv1937973469:84 , srv1939220588:84 , srv196359457:84 , srv2092741006:84 , srv279423674:84 , srv372461097:84 , srv470489846:84 , srv481924240:84 , srv488609698:84 , srv56717665:84 , srv568594944:84 , srv645821754:84 , srv716670966:84 , srv725354524:84 , srv792090647:84 , srv94317885:84 , srv1472682810:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-08-24 13:58:04,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-24 13:58:04,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv56717665=23, srv1844091921=9, srv470489846=20, srv196359457=16, srv1472682810=2, srv1701149567=6, srv190245973=13, srv1159796009=1, srv645821754=25, srv792090647=28, srv1013795781=0, srv1542331641=4, srv1937973469=14, srv2092741006=17, srv149360983=3, srv725354524=27, srv1860933127=12, srv1744780700=7, srv279423674=18, srv716670966=26, srv1854703096=11, srv372461097=19, srv168433757=5, srv1846002324=10, srv94317885=29, srv488609698=22, srv568594944=24, srv481924240=21, srv1778914283=8, srv1939220588=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-24 13:58:04,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-24 13:58:04,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-24 13:58:04,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-24 13:58:04,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-24 13:58:04,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-24 13:58:04,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-24 13:58:04,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-24 13:58:04,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-24 13:58:04,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-24 13:58:04,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-24 13:58:04,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-24 13:58:04,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-24 13:58:04,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-24 13:58:04,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-24 13:58:04,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-24 13:58:04,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-24 13:58:04,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-24 13:58:04,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-24 13:58:04,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-24 13:58:04,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-24 13:58:04,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-24 13:58:04,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-24 13:58:04,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-24 13:58:04,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-24 13:58:04,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-24 13:58:04,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-24 13:58:04,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-24 13:58:04,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-24 13:58:04,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-24 13:58:04,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-24 13:58:04,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-24 13:58:04,262 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1451732501532126, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-24 13:58:04,262 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912650479435169, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1451732501532126, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-08-24 14:00:04,241 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5554423 different iterations. Found a solution that moves 1854 regions; Going from a computed imbalance of 0.9912650479435169 to a new imbalance of 4.3273284898463826E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6866666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-24 14:00:04,250 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1013795781:90 , srv1159796009:90 , srv1472682810:90 , srv149360983:90 , srv1542331641:90 , srv168433757:90 , srv1701149567:90 , srv1744780700:90 , srv1778914283:90 , srv1844091921:90 , srv1846002324:90 , srv1854703096:90 , srv1860933127:90 , srv190245973:90 , srv1937973469:90 , srv1939220588:90 , srv196359457:90 , srv2092741006:90 , srv279423674:90 , srv372461097:90 , srv470489846:90 , srv481924240:90 , srv488609698:90 , srv56717665:90 , srv568594944:90 , srv645821754:90 , srv716670966:90 , srv725354524:90 , srv792090647:90 , srv94317885:90 } 2023-08-24 14:00:04,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-24 14:00:04,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv56717665=23, srv1844091921=9, srv470489846=20, srv196359457=16, srv1472682810=2, srv1701149567=6, srv190245973=13, srv1159796009=1, srv645821754=25, srv792090647=28, srv1013795781=0, srv1542331641=4, srv1937973469=14, srv2092741006=17, srv149360983=3, srv725354524=27, srv1860933127=12, srv1744780700=7, srv279423674=18, srv716670966=26, srv1854703096=11, srv372461097=19, srv168433757=5, srv1846002324=10, srv94317885=29, srv488609698=22, srv568594944=24, srv481924240=21, srv1778914283=8, srv1939220588=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-24 14:00:04,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-24 14:00:04,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-24 14:00:04,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-24 14:00:04,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-24 14:00:04,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-24 14:00:04,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-24 14:00:04,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-24 14:00:04,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-24 14:00:04,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-24 14:00:04,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-24 14:00:04,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-24 14:00:04,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-24 14:00:04,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-24 14:00:04,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-24 14:00:04,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-24 14:00:04,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-24 14:00:04,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-24 14:00:04,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-24 14:00:04,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-24 14:00:04,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-24 14:00:04,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-24 14:00:04,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-24 14:00:04,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-24 14:00:04,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-24 14:00:04,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-24 14:00:04,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-24 14:00:04,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-24 14:00:04,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-24 14:00:04,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-24 14:00:04,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-24 14:00:04,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-24 14:00:04,271 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-24 14:00:04,272 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1013795781:90 , srv1159796009:90 , srv1472682810:90 , srv149360983:90 , srv1542331641:90 , srv168433757:90 , srv1701149567:90 , srv1744780700:90 , srv1778914283:90 , srv1844091921:90 , srv1846002324:90 , srv1854703096:90 , srv1860933127:90 , srv190245973:90 , srv1937973469:90 , srv1939220588:90 , srv196359457:90 , srv2092741006:90 , srv279423674:90 , srv372461097:90 , srv470489846:90 , srv481924240:90 , srv488609698:90 , srv56717665:90 , srv568594944:90 , srv645821754:90 , srv716670966:90 , srv725354524:90 , srv792090647:90 , srv94317885:90 } 2023-08-24 14:00:04,288 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=236 (was 327), ProcessCount=176 (was 172) - ProcessCount LEAK? -, AvailableMemoryMB=10828 (was 10722) - AvailableMemoryMB LEAK? -