2023-02-24 18:53:56,322 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-02-24 18:53:56,643 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-24 18:53:56,763 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-24 18:53:56,836 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-24 18:53:56,847 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-24 18:53:56,867 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=347, ProcessCount=171, AvailableMemoryMB=6559 2023-02-24 18:53:56,870 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-24 18:53:56,870 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-24 18:53:57,024 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1023584840:84 , srv1043766822:84 , srv1048342159:84 , srv1074032775:84 , srv1091572319:84 , srv1126581485:84 , srv1139130635:84 , srv1164812150:84 , srv121677622:84 , srv1287323990:84 , srv1307509847:84 , srv1339443792:84 , srv1360669737:84 , srv1367650570:84 , srv1369157131:84 , srv1380370040:84 , srv1381613319:84 , srv1390751201:84 , srv1472384535:84 , srv1494471759:84 , srv1547672429:84 , srv1587245812:84 , srv1592319761:84 , srv1597122513:84 , srv1597941916:84 , srv1619877781:84 , srv1621927943:84 , srv1649760030:84 , srv1662650376:84 , srv1665816852:84 , srv1712099139:84 , srv1742044892:84 , srv1798318090:84 , srv1808079066:84 , srv18261986:84 , srv185270833:84 , srv1860215688:84 , srv1866692491:84 , srv1869856841:84 , srv1879841384:84 , srv1881193288:84 , srv1903764624:84 , srv1908831139:84 , srv1927401991:84 , srv1960961942:84 , srv1973534617:84 , srv1975938425:84 , srv2010564722:84 , srv2055480293:84 , srv2066327066:84 , srv2072203922:84 , srv2085304884:84 , srv2108356633:84 , srv2109090951:84 , srv2122891065:84 , srv2146857799:84 , srv215950180:84 , srv2308066:84 , srv244307830:84 , srv259432567:84 , srv275909546:84 , srv281566902:84 , srv347311582:84 , srv360375322:84 , srv397219910:84 , srv405534247:84 , srv414349787:84 , srv415699919:84 , srv416147470:84 , srv422857068:84 , srv442300169:84 , srv489311493:84 , srv498792760:84 , srv508266332:84 , srv547030520:84 , srv571061939:84 , srv57233364:84 , srv575988847:84 , srv585104548:84 , srv586240875:84 , srv626516098:84 , srv690441477:84 , srv705680376:84 , srv724049186:84 , srv734066907:84 , srv734188659:84 , srv742617979:84 , srv762440166:84 , srv767059886:84 , srv770685906:84 , srv837652972:84 , srv839028719:84 , srv848019054:84 , srv85163490:84 , srv908832985:84 , srv909140491:84 , srv958094748:84 , srv964302366:84 , srv979375367:84 , srv1561882887:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-02-24 18:53:57,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-24 18:53:57,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1381613319=16, srv586240875=80, srv1367650570=13, srv1662650376=29, srv1808079066=34, srv1164812150=7, srv767059886=89, srv734188659=86, srv1649760030=28, srv571061939=76, srv734066907=85, srv405534247=66, srv839028719=92, srv2010564722=48, srv724049186=84, srv705680376=83, srv1091572319=4, srv1619877781=26, srv2085304884=52, srv1927401991=44, srv416147470=69, srv964302366=98, srv414349787=67, srv2146857799=56, srv415699919=68, srv979375367=99, srv1973534617=46, srv275909546=61, srv742617979=87, srv1023584840=0, srv1742044892=32, srv547030520=75, srv585104548=79, srv121677622=8, srv244307830=59, srv690441477=82, srv1665816852=30, srv848019054=93, srv259432567=60, srv397219910=65, srv498792760=73, srv1903764624=42, srv1139130635=6, srv185270833=36, srv1960961942=45, srv1561882887=21, srv2108356633=53, srv2072203922=51, srv489311493=72, srv1048342159=2, srv762440166=88, srv85163490=94, srv1860215688=37, srv2055480293=49, srv1597941916=25, srv958094748=97, srv2308066=58, srv1287323990=9, srv1380370040=15, srv1597122513=24, srv1472384535=18, srv1712099139=31, srv1494471759=19, srv18261986=35, srv442300169=71, srv1869856841=39, srv1360669737=12, srv1126581485=5, srv57233364=77, srv575988847=78, srv770685906=90, srv1043766822=1, srv281566902=62, srv1881193288=41, srv1390751201=17, srv1369157131=14, srv1798318090=33, srv837652972=91, srv1879841384=40, srv2122891065=55, srv2109090951=54, srv1339443792=11, srv422857068=70, srv1621927943=27, srv1908831139=43, srv1074032775=3, srv1592319761=23, srv1866692491=38, srv909140491=96, srv508266332=74, srv360375322=64, srv1307509847=10, srv2066327066=50, srv626516098=81, srv908832985=95, srv215950180=57, srv1975938425=47, srv1547672429=20, srv1587245812=22, srv347311582=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-24 18:53:57,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-24 18:53:57,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-24 18:53:57,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-24 18:53:57,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-24 18:53:57,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-24 18:53:57,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-24 18:53:57,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-24 18:53:57,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-24 18:53:57,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-24 18:53:57,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-24 18:53:57,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-24 18:53:57,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-24 18:53:57,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-24 18:53:57,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-24 18:53:57,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-24 18:53:57,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-24 18:53:57,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-24 18:53:57,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-24 18:53:57,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-24 18:53:57,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-24 18:53:57,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-24 18:53:57,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-24 18:53:57,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-24 18:53:57,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-24 18:53:57,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-24 18:53:57,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-24 18:53:57,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-24 18:53:57,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-24 18:53:57,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-24 18:53:57,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-24 18:53:57,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-24 18:53:57,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-24 18:53:57,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-24 18:53:57,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-24 18:53:57,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-24 18:53:57,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-24 18:53:57,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-24 18:53:57,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-24 18:53:57,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-24 18:53:57,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-24 18:53:57,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-24 18:53:57,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-24 18:53:57,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-24 18:53:57,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-24 18:53:57,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-24 18:53:57,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-24 18:53:57,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-24 18:53:57,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-24 18:53:57,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-24 18:53:57,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-24 18:53:57,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-24 18:53:57,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-24 18:53:57,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-24 18:53:57,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-24 18:53:57,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-24 18:53:57,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-24 18:53:57,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-24 18:53:57,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-24 18:53:57,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-24 18:53:57,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-24 18:53:57,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-24 18:53:57,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-24 18:53:57,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-24 18:53:57,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-24 18:53:57,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-24 18:53:57,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-24 18:53:57,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-24 18:53:57,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-24 18:53:57,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-24 18:53:57,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-24 18:53:57,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-24 18:53:57,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-24 18:53:57,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-24 18:53:57,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-24 18:53:57,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-24 18:53:57,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-24 18:53:57,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-24 18:53:57,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-24 18:53:57,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-24 18:53:57,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-24 18:53:57,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-24 18:53:57,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-24 18:53:57,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-24 18:53:57,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-24 18:53:57,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-24 18:53:57,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-24 18:53:57,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-24 18:53:57,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-24 18:53:57,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-24 18:53:57,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-24 18:53:57,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-24 18:53:57,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-24 18:53:57,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-24 18:53:57,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-24 18:53:57,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-24 18:53:57,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-24 18:53:57,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-24 18:53:57,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-24 18:53:57,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-24 18:53:57,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-24 18:53:57,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-24 18:53:57,483 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-24 18:53:57,484 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-24 18:53:57,484 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-24 18:53:58,203 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 995 ms to try 5000 different iterations. Found a solution that moves 4711 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17443814066521598. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039394025727264456, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05598420492333411, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5234444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039394025727264456, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16975, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23483333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-24 18:53:58,233 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1547672429:82 , srv360375322:82 , srv415699919:82 , srv1139130635:83 , srv1287323990:83 , srv1881193288:83 , srv2055480293:83 , srv2072203922:83 , srv281566902:83 , srv848019054:83 , srv85163490:83 , srv1339443792:84 , srv1367650570:84 , srv1621927943:84 , srv1712099139:84 , srv1866692491:84 , srv1960961942:84 , srv1975938425:84 , srv259432567:84 , srv837652972:84 , srv1023584840:85 , srv1307509847:85 , srv1472384535:85 , srv1587245812:85 , srv1869856841:85 , srv1973534617:85 , srv2308066:85 , srv405534247:85 , srv414349787:85 , srv547030520:85 , srv767059886:85 , srv1048342159:86 , srv1074032775:86 , srv1091572319:86 , srv1360669737:86 , srv1597122513:86 , srv1665816852:86 , srv1742044892:86 , srv1798318090:86 , srv185270833:86 , srv1860215688:86 , srv1879841384:86 , srv1908831139:86 , srv1927401991:86 , srv2066327066:86 , srv2085304884:86 , srv2108356633:86 , srv2122891065:86 , srv244307830:86 , srv275909546:86 , srv397219910:86 , srv422857068:86 , srv498792760:86 , srv508266332:86 , srv586240875:86 , srv690441477:86 , srv724049186:86 , srv762440166:86 , srv839028719:86 , srv909140491:86 , srv1126581485:87 , srv1164812150:87 , srv121677622:87 , srv1369157131:87 , srv1381613319:87 , srv1390751201:87 , srv1494471759:87 , srv1903764624:87 , srv2010564722:87 , srv2109090951:87 , srv215950180:87 , srv57233364:87 , srv908832985:87 , srv1619877781:88 , srv1662650376:88 , srv1808079066:88 , srv2146857799:88 , srv416147470:88 , srv489311493:88 , srv571061939:88 , srv585104548:88 , srv705680376:88 , srv734188659:88 , srv742617979:88 , srv1043766822:89 , srv1649760030:89 , srv442300169:89 , srv575988847:89 , srv734066907:89 , srv979375367:89 , srv18261986:90 , srv626516098:90 , srv770685906:90 , srv1592319761:91 , srv347311582:91 , srv958094748:91 , srv1380370040:94 , srv964302366:97 , srv1597941916:98 , srv1561882887:440 } 2023-02-24 18:53:58,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-24 18:53:58,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1381613319=16, srv586240875=80, srv1367650570=13, srv1662650376=29, srv1808079066=34, srv1164812150=7, srv767059886=89, srv734188659=86, srv1649760030=28, srv571061939=76, srv734066907=85, srv405534247=66, srv839028719=92, srv2010564722=48, srv724049186=84, srv705680376=83, srv1091572319=4, srv1619877781=26, srv2085304884=52, srv1927401991=44, srv416147470=69, srv964302366=98, srv414349787=67, srv2146857799=56, srv415699919=68, srv979375367=99, srv1973534617=46, srv275909546=61, srv742617979=87, srv1023584840=0, srv1742044892=32, srv547030520=75, srv585104548=79, srv121677622=8, srv244307830=59, srv690441477=82, srv1665816852=30, srv848019054=93, srv259432567=60, srv397219910=65, srv498792760=73, srv1903764624=42, srv1139130635=6, srv185270833=36, srv1960961942=45, srv1561882887=21, srv2108356633=53, srv2072203922=51, srv489311493=72, srv1048342159=2, srv762440166=88, srv85163490=94, srv1860215688=37, srv2055480293=49, srv1597941916=25, srv958094748=97, srv2308066=58, srv1287323990=9, srv1380370040=15, srv1597122513=24, srv1472384535=18, srv1712099139=31, srv1494471759=19, srv18261986=35, srv442300169=71, srv1869856841=39, srv1360669737=12, srv1126581485=5, srv57233364=77, srv575988847=78, srv770685906=90, srv1043766822=1, srv281566902=62, srv1881193288=41, srv1390751201=17, srv1369157131=14, srv1798318090=33, srv837652972=91, srv1879841384=40, srv2122891065=55, srv2109090951=54, srv1339443792=11, srv422857068=70, srv1621927943=27, srv1908831139=43, srv1074032775=3, srv1592319761=23, srv1866692491=38, srv909140491=96, srv508266332=74, srv360375322=64, srv1307509847=10, srv2066327066=50, srv626516098=81, srv908832985=95, srv215950180=57, srv1975938425=47, srv1547672429=20, srv1587245812=22, srv347311582=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-24 18:53:58,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-24 18:53:58,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-24 18:53:58,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-24 18:53:58,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-24 18:53:58,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-24 18:53:58,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-24 18:53:58,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-24 18:53:58,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-24 18:53:58,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-24 18:53:58,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-24 18:53:58,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-24 18:53:58,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-24 18:53:58,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-24 18:53:58,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-24 18:53:58,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-24 18:53:58,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-24 18:53:58,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-24 18:53:58,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-24 18:53:58,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-24 18:53:58,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-24 18:53:58,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-24 18:53:58,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-24 18:53:58,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-24 18:53:58,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-24 18:53:58,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-24 18:53:58,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-24 18:53:58,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-24 18:53:58,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-24 18:53:58,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-24 18:53:58,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-24 18:53:58,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-24 18:53:58,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-24 18:53:58,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-24 18:53:58,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-24 18:53:58,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-24 18:53:58,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-24 18:53:58,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-24 18:53:58,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-24 18:53:58,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-24 18:53:58,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-24 18:53:58,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-24 18:53:58,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-24 18:53:58,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-24 18:53:58,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-24 18:53:58,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-24 18:53:58,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-24 18:53:58,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-24 18:53:58,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-24 18:53:58,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-24 18:53:58,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-24 18:53:58,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-24 18:53:58,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-24 18:53:58,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-24 18:53:58,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-24 18:53:58,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-24 18:53:58,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-24 18:53:58,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-24 18:53:58,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-24 18:53:58,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-24 18:53:58,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-24 18:53:58,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-24 18:53:58,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-24 18:53:58,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-24 18:53:58,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-24 18:53:58,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-24 18:53:58,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-24 18:53:58,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-24 18:53:58,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-24 18:53:58,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-24 18:53:58,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-24 18:53:58,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-24 18:53:58,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-24 18:53:58,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-24 18:53:58,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-24 18:53:58,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-24 18:53:58,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-24 18:53:58,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-24 18:53:58,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-24 18:53:58,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-24 18:53:58,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-24 18:53:58,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-24 18:53:58,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-24 18:53:58,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-24 18:53:58,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-24 18:53:58,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-24 18:53:58,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-24 18:53:58,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-24 18:53:58,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-24 18:53:58,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-24 18:53:58,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-24 18:53:58,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-24 18:53:58,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-24 18:53:58,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-24 18:53:58,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-24 18:53:58,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-24 18:53:58,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-24 18:53:58,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-24 18:53:58,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-24 18:53:58,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-24 18:53:58,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-24 18:53:58,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-24 18:53:58,508 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039394025727264456, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05598420492333411, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039394025727264456, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16975, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23483333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-24 18:53:58,508 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-24 18:53:58,509 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.174405153538168, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039394025727264456, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05598420492333411, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039394025727264456, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16975, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23483333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-24 18:53:58,929 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 614 ms to try 5000 different iterations. Found a solution that moves 2595 regions; Going from a computed imbalance of 0.174405153538168 to a new imbalance of 0.0163735992128562. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02104395959511489); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03424018943274953, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28833333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02104395959511489); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0455, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-24 18:53:58,943 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1597941916:75 , srv1380370040:78 , srv958094748:79 , srv1592319761:80 , srv18261986:81 , srv347311582:81 , srv964302366:81 , srv1649760030:82 , srv121677622:83 , srv1662650376:83 , srv2109090951:83 , srv2146857799:83 , srv442300169:83 , srv575988847:83 , srv585104548:83 , srv626516098:83 , srv734188659:83 , srv1043766822:84 , srv1369157131:84 , srv215950180:84 , srv571061939:84 , srv705680376:84 , srv742617979:84 , srv770685906:84 , srv1164812150:85 , srv1494471759:85 , srv1587245812:85 , srv1597122513:85 , srv1879841384:85 , srv1903764624:85 , srv2085304884:85 , srv2108356633:85 , srv2122891065:85 , srv275909546:85 , srv508266332:85 , srv734066907:85 , srv767059886:85 , srv839028719:85 , srv909140491:85 , srv1048342159:86 , srv1074032775:86 , srv1091572319:86 , srv1126581485:86 , srv1390751201:86 , srv1619877781:86 , srv1621927943:86 , srv1742044892:86 , srv1869856841:86 , srv1973534617:86 , srv397219910:86 , srv414349787:86 , srv416147470:86 , srv422857068:86 , srv498792760:86 , srv547030520:86 , srv57233364:86 , srv586240875:86 , srv724049186:86 , srv762440166:86 , srv908832985:86 , srv979375367:86 , srv1287323990:87 , srv1307509847:87 , srv1339443792:87 , srv1360669737:87 , srv1367650570:87 , srv1381613319:87 , srv1472384535:87 , srv1665816852:87 , srv1712099139:87 , srv1798318090:87 , srv1808079066:87 , srv1860215688:87 , srv1881193288:87 , srv1908831139:87 , srv1975938425:87 , srv2010564722:87 , srv2066327066:87 , srv2308066:87 , srv244307830:87 , srv281566902:87 , srv405534247:87 , srv489311493:87 , srv1023584840:88 , srv1927401991:88 , srv1960961942:88 , srv259432567:88 , srv690441477:88 , srv848019054:88 , srv185270833:89 , srv837652972:89 , srv85163490:89 , srv1139130635:90 , srv1547672429:90 , srv1866692491:90 , srv2055480293:90 , srv2072203922:90 , srv415699919:90 , srv360375322:91 , srv1561882887:521 } 2023-02-24 18:53:58,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-24 18:53:58,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1381613319=16, srv586240875=80, srv1367650570=13, srv1662650376=29, srv1808079066=34, srv1164812150=7, srv767059886=89, srv734188659=86, srv1649760030=28, srv571061939=76, srv734066907=85, srv405534247=66, srv839028719=92, srv2010564722=48, srv724049186=84, srv705680376=83, srv1091572319=4, srv1619877781=26, srv2085304884=52, srv1927401991=44, srv416147470=69, srv964302366=98, srv414349787=67, srv2146857799=56, srv415699919=68, srv979375367=99, srv1973534617=46, srv275909546=61, srv742617979=87, srv1023584840=0, srv1742044892=32, srv547030520=75, srv585104548=79, srv121677622=8, srv244307830=59, srv690441477=82, srv1665816852=30, srv848019054=93, srv259432567=60, srv397219910=65, srv498792760=73, srv1903764624=42, srv1139130635=6, srv185270833=36, srv1960961942=45, srv1561882887=21, srv2108356633=53, srv2072203922=51, srv489311493=72, srv1048342159=2, srv762440166=88, srv85163490=94, srv1860215688=37, srv2055480293=49, srv1597941916=25, srv958094748=97, srv2308066=58, srv1287323990=9, srv1380370040=15, srv1597122513=24, srv1472384535=18, srv1712099139=31, srv1494471759=19, srv18261986=35, srv442300169=71, srv1869856841=39, srv1360669737=12, srv1126581485=5, srv57233364=77, srv575988847=78, srv770685906=90, srv1043766822=1, srv281566902=62, srv1881193288=41, srv1390751201=17, srv1369157131=14, srv1798318090=33, srv837652972=91, srv1879841384=40, srv2122891065=55, srv2109090951=54, srv1339443792=11, srv422857068=70, srv1621927943=27, srv1908831139=43, srv1074032775=3, srv1592319761=23, srv1866692491=38, srv909140491=96, srv508266332=74, srv360375322=64, srv1307509847=10, srv2066327066=50, srv626516098=81, srv908832985=95, srv215950180=57, srv1975938425=47, srv1547672429=20, srv1587245812=22, srv347311582=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-24 18:53:59,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-24 18:53:59,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-24 18:53:59,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-24 18:53:59,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-24 18:53:59,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-24 18:53:59,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-24 18:53:59,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-24 18:53:59,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-24 18:53:59,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-24 18:53:59,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-24 18:53:59,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-24 18:53:59,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-24 18:53:59,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-24 18:53:59,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-24 18:53:59,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-24 18:53:59,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-24 18:53:59,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-24 18:53:59,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-24 18:53:59,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-24 18:53:59,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-24 18:53:59,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-24 18:53:59,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-24 18:53:59,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-24 18:53:59,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-24 18:53:59,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-24 18:53:59,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-24 18:53:59,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-24 18:53:59,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-24 18:53:59,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-24 18:53:59,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-24 18:53:59,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-24 18:53:59,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-24 18:53:59,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-24 18:53:59,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-24 18:53:59,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-24 18:53:59,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-24 18:53:59,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-24 18:53:59,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-24 18:53:59,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-24 18:53:59,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-24 18:53:59,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-24 18:53:59,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-24 18:53:59,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-24 18:53:59,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-24 18:53:59,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-24 18:53:59,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-24 18:53:59,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-24 18:53:59,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-24 18:53:59,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-24 18:53:59,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-24 18:53:59,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-24 18:53:59,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-24 18:53:59,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-24 18:53:59,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-24 18:53:59,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-24 18:53:59,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-24 18:53:59,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-24 18:53:59,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-24 18:53:59,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-24 18:53:59,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-24 18:53:59,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-24 18:53:59,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-24 18:53:59,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-24 18:53:59,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-24 18:53:59,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-24 18:53:59,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-24 18:53:59,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-24 18:53:59,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-24 18:53:59,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-24 18:53:59,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-24 18:53:59,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-24 18:53:59,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-24 18:53:59,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-24 18:53:59,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-24 18:53:59,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-24 18:53:59,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-24 18:53:59,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-24 18:53:59,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-24 18:53:59,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-24 18:53:59,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-24 18:53:59,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-24 18:53:59,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-24 18:53:59,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-24 18:53:59,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-24 18:53:59,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-24 18:53:59,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-24 18:53:59,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-24 18:53:59,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-24 18:53:59,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-24 18:53:59,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-24 18:53:59,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-24 18:53:59,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-24 18:53:59,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-24 18:53:59,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-24 18:53:59,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-24 18:53:59,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-24 18:53:59,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-24 18:53:59,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-24 18:53:59,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-24 18:53:59,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-24 18:53:59,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-24 18:53:59,198 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02104395959511489); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03424018943274953, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02104395959511489); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0455, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-24 18:53:59,198 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-24 18:53:59,198 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01635542863448864, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02104395959511489); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03424018943274953, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02104395959511489); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0455, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-24 18:53:59,461 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 407 ms to try 5000 different iterations. Found a solution that moves 988 regions; Going from a computed imbalance of 0.01635542863448864 to a new imbalance of 8.311184334927E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00421104314862138); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008593579416223516); MoveCostFunction : (multiplier=7.0, imbalance=0.10977777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00421104314862138); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-24 18:53:59,465 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv964302366:80 , srv1808079066:83 , srv185270833:83 , srv489311493:83 , srv690441477:83 , srv734066907:83 , srv979375367:83 , srv1023584840:84 , srv1126581485:84 , srv1139130635:84 , srv1381613319:84 , srv1619877781:84 , srv1665816852:84 , srv1866692491:84 , srv1908831139:84 , srv1927401991:84 , srv244307830:84 , srv416147470:84 , srv57233364:84 , srv626516098:84 , srv770685906:84 , srv837652972:84 , srv1048342159:85 , srv1074032775:85 , srv1091572319:85 , srv1360669737:85 , srv1380370040:85 , srv1390751201:85 , srv1494471759:85 , srv1597941916:85 , srv1742044892:85 , srv1798318090:85 , srv1860215688:85 , srv1960961942:85 , srv2055480293:85 , srv2066327066:85 , srv2072203922:85 , srv347311582:85 , srv360375322:85 , srv405534247:85 , srv422857068:85 , srv442300169:85 , srv586240875:85 , srv724049186:85 , srv742617979:85 , srv908832985:85 , srv1043766822:86 , srv1164812150:86 , srv1307509847:86 , srv1339443792:86 , srv1367650570:86 , srv1369157131:86 , srv1472384535:86 , srv1547672429:86 , srv1592319761:86 , srv1597122513:86 , srv1649760030:86 , srv18261986:86 , srv1869856841:86 , srv1879841384:86 , srv1903764624:86 , srv2010564722:86 , srv2122891065:86 , srv215950180:86 , srv2308066:86 , srv259432567:86 , srv397219910:86 , srv414349787:86 , srv415699919:86 , srv498792760:86 , srv508266332:86 , srv571061939:86 , srv575988847:86 , srv585104548:86 , srv705680376:86 , srv734188659:86 , srv848019054:86 , srv909140491:86 , srv1621927943:87 , srv1712099139:87 , srv1881193288:87 , srv1973534617:87 , srv1975938425:87 , srv2085304884:87 , srv2108356633:87 , srv2146857799:87 , srv275909546:87 , srv547030520:87 , srv762440166:87 , srv839028719:87 , srv85163490:87 , srv121677622:88 , srv1287323990:88 , srv1587245812:88 , srv1662650376:88 , srv2109090951:88 , srv281566902:88 , srv767059886:88 , srv958094748:89 , srv1561882887:534 } 2023-02-24 18:53:59,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-24 18:53:59,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1381613319=16, srv586240875=80, srv1367650570=13, srv1662650376=29, srv1808079066=34, srv1164812150=7, srv767059886=89, srv734188659=86, srv1649760030=28, srv571061939=76, srv734066907=85, srv405534247=66, srv839028719=92, srv2010564722=48, srv724049186=84, srv705680376=83, srv1091572319=4, srv1619877781=26, srv2085304884=52, srv1927401991=44, srv416147470=69, srv964302366=98, srv414349787=67, srv2146857799=56, srv415699919=68, srv979375367=99, srv1973534617=46, srv275909546=61, srv742617979=87, srv1023584840=0, srv1742044892=32, srv547030520=75, srv585104548=79, srv121677622=8, srv244307830=59, srv690441477=82, srv1665816852=30, srv848019054=93, srv259432567=60, srv397219910=65, srv498792760=73, srv1903764624=42, srv1139130635=6, srv185270833=36, srv1960961942=45, srv1561882887=21, srv2108356633=53, srv2072203922=51, srv489311493=72, srv1048342159=2, srv762440166=88, srv85163490=94, srv1860215688=37, srv2055480293=49, srv1597941916=25, srv958094748=97, srv2308066=58, srv1287323990=9, srv1380370040=15, srv1597122513=24, srv1472384535=18, srv1712099139=31, srv1494471759=19, srv18261986=35, srv442300169=71, srv1869856841=39, srv1360669737=12, srv1126581485=5, srv57233364=77, srv575988847=78, srv770685906=90, srv1043766822=1, srv281566902=62, srv1881193288=41, srv1390751201=17, srv1369157131=14, srv1798318090=33, srv837652972=91, srv1879841384=40, srv2122891065=55, srv2109090951=54, srv1339443792=11, srv422857068=70, srv1621927943=27, srv1908831139=43, srv1074032775=3, srv1592319761=23, srv1866692491=38, srv909140491=96, srv508266332=74, srv360375322=64, srv1307509847=10, srv2066327066=50, srv626516098=81, srv908832985=95, srv215950180=57, srv1975938425=47, srv1547672429=20, srv1587245812=22, srv347311582=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-24 18:53:59,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-24 18:53:59,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-24 18:53:59,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-24 18:53:59,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-24 18:53:59,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-24 18:53:59,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-24 18:53:59,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-24 18:53:59,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-24 18:53:59,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-24 18:53:59,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-24 18:53:59,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-24 18:53:59,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-24 18:53:59,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-24 18:53:59,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-24 18:53:59,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-24 18:53:59,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-24 18:53:59,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-24 18:53:59,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-24 18:53:59,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-24 18:53:59,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-24 18:53:59,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-24 18:53:59,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-24 18:53:59,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-24 18:53:59,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-24 18:53:59,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-24 18:53:59,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-24 18:53:59,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-24 18:53:59,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-24 18:53:59,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-24 18:53:59,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-24 18:53:59,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-24 18:53:59,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-24 18:53:59,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-24 18:53:59,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-24 18:53:59,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-24 18:53:59,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-24 18:53:59,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-24 18:53:59,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-24 18:53:59,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-24 18:53:59,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-24 18:53:59,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-24 18:53:59,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-24 18:53:59,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-24 18:53:59,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-24 18:53:59,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-24 18:53:59,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-24 18:53:59,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-24 18:53:59,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-24 18:53:59,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-24 18:53:59,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-24 18:53:59,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-24 18:53:59,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-24 18:53:59,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-24 18:53:59,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-24 18:53:59,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-24 18:53:59,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-24 18:53:59,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-24 18:53:59,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-24 18:53:59,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-24 18:53:59,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-24 18:53:59,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-24 18:53:59,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-24 18:53:59,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-24 18:53:59,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-24 18:53:59,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-24 18:53:59,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-24 18:53:59,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-24 18:53:59,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-24 18:53:59,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-24 18:53:59,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-24 18:53:59,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-24 18:53:59,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-24 18:53:59,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-24 18:53:59,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-24 18:53:59,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-24 18:53:59,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-24 18:53:59,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-24 18:53:59,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-24 18:53:59,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-24 18:53:59,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-24 18:53:59,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-24 18:53:59,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-24 18:53:59,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-24 18:53:59,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-24 18:53:59,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-24 18:53:59,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-24 18:53:59,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-24 18:53:59,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-24 18:53:59,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-24 18:53:59,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-24 18:53:59,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-24 18:53:59,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-24 18:53:59,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-24 18:53:59,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-24 18:53:59,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-24 18:53:59,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-24 18:53:59,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-24 18:53:59,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-24 18:53:59,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-24 18:53:59,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-24 18:53:59,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-24 18:53:59,636 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00421104314862138); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008593579416223516); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00421104314862138); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-24 18:53:59,637 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-24 18:53:59,637 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.242003096286737E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00421104314862138); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008593579416223516); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00421104314862138); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-24 18:53:59,976 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 458 ms to try 5000 different iterations. Found a solution that moves 258 regions; Going from a computed imbalance of 8.242003096286737E-4 to a new imbalance of 4.1946898974814634E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.028666666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-24 18:53:59,977 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2010564722:82 , srv1043766822:83 , srv1390751201:83 , srv1662650376:83 , srv1712099139:83 , srv1798318090:83 , srv2066327066:83 , srv275909546:83 , srv547030520:83 , srv705680376:83 , srv762440166:83 , srv85163490:83 , srv958094748:83 , srv1164812150:84 , srv121677622:84 , srv1287323990:84 , srv1307509847:84 , srv1360669737:84 , srv1381613319:84 , srv1472384535:84 , srv1587245812:84 , srv1597941916:84 , srv1619877781:84 , srv1866692491:84 , srv1881193288:84 , srv1927401991:84 , srv1973534617:84 , srv1975938425:84 , srv2055480293:84 , srv2072203922:84 , srv2085304884:84 , srv2108356633:84 , srv2109090951:84 , srv2146857799:84 , srv2308066:84 , srv259432567:84 , srv281566902:84 , srv360375322:84 , srv397219910:84 , srv415699919:84 , srv489311493:84 , srv498792760:84 , srv571061939:84 , srv575988847:84 , srv767059886:84 , srv770685906:84 , srv839028719:84 , srv908832985:84 , srv964302366:84 , srv979375367:84 , srv1023584840:85 , srv1048342159:85 , srv1074032775:85 , srv1091572319:85 , srv1126581485:85 , srv1139130635:85 , srv1339443792:85 , srv1367650570:85 , srv1369157131:85 , srv1380370040:85 , srv1494471759:85 , srv1547672429:85 , srv1592319761:85 , srv1597122513:85 , srv1621927943:85 , srv1649760030:85 , srv1742044892:85 , srv1808079066:85 , srv18261986:85 , srv185270833:85 , srv1860215688:85 , srv1869856841:85 , srv1879841384:85 , srv1903764624:85 , srv1908831139:85 , srv1960961942:85 , srv2122891065:85 , srv215950180:85 , srv244307830:85 , srv347311582:85 , srv405534247:85 , srv414349787:85 , srv416147470:85 , srv422857068:85 , srv442300169:85 , srv508266332:85 , srv57233364:85 , srv585104548:85 , srv586240875:85 , srv626516098:85 , srv690441477:85 , srv724049186:85 , srv734066907:85 , srv734188659:85 , srv837652972:85 , srv848019054:85 , srv909140491:85 , srv1665816852:86 , srv742617979:86 , srv1561882887:647 } 2023-02-24 18:53:59,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-24 18:53:59,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1381613319=16, srv586240875=80, srv1367650570=13, srv1662650376=29, srv1808079066=34, srv1164812150=7, srv767059886=89, srv734188659=86, srv1649760030=28, srv571061939=76, srv734066907=85, srv405534247=66, srv839028719=92, srv2010564722=48, srv724049186=84, srv705680376=83, srv1091572319=4, srv1619877781=26, srv2085304884=52, srv1927401991=44, srv416147470=69, srv964302366=98, srv414349787=67, srv2146857799=56, srv415699919=68, srv979375367=99, srv1973534617=46, srv275909546=61, srv742617979=87, srv1023584840=0, srv1742044892=32, srv547030520=75, srv585104548=79, srv121677622=8, srv244307830=59, srv690441477=82, srv1665816852=30, srv848019054=93, srv259432567=60, srv397219910=65, srv498792760=73, srv1903764624=42, srv1139130635=6, srv185270833=36, srv1960961942=45, srv1561882887=21, srv2108356633=53, srv2072203922=51, srv489311493=72, srv1048342159=2, srv762440166=88, srv85163490=94, srv1860215688=37, srv2055480293=49, srv1597941916=25, srv958094748=97, srv2308066=58, srv1287323990=9, srv1380370040=15, srv1597122513=24, srv1472384535=18, srv1712099139=31, srv1494471759=19, srv18261986=35, srv442300169=71, srv1869856841=39, srv1360669737=12, srv1126581485=5, srv57233364=77, srv575988847=78, srv770685906=90, srv1043766822=1, srv281566902=62, srv1881193288=41, srv1390751201=17, srv1369157131=14, srv1798318090=33, srv837652972=91, srv1879841384=40, srv2122891065=55, srv2109090951=54, srv1339443792=11, srv422857068=70, srv1621927943=27, srv1908831139=43, srv1074032775=3, srv1592319761=23, srv1866692491=38, srv909140491=96, srv508266332=74, srv360375322=64, srv1307509847=10, srv2066327066=50, srv626516098=81, srv908832985=95, srv215950180=57, srv1975938425=47, srv1547672429=20, srv1587245812=22, srv347311582=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-24 18:53:59,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-24 18:53:59,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-24 18:53:59,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-24 18:53:59,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-24 18:53:59,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-24 18:53:59,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-24 18:53:59,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-24 18:53:59,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-24 18:53:59,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-24 18:53:59,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-24 18:53:59,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-24 18:53:59,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-24 18:53:59,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-24 18:53:59,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-24 18:53:59,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-24 18:53:59,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-24 18:53:59,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-24 18:53:59,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-24 18:53:59,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-24 18:53:59,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-24 18:53:59,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-24 18:53:59,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-24 18:53:59,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-24 18:53:59,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-24 18:53:59,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-24 18:53:59,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-24 18:53:59,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-24 18:53:59,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-24 18:53:59,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-24 18:53:59,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-24 18:53:59,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-24 18:53:59,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-24 18:53:59,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-24 18:53:59,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-24 18:53:59,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-24 18:53:59,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-24 18:53:59,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-24 18:53:59,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-24 18:53:59,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-24 18:53:59,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-24 18:53:59,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-24 18:53:59,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-24 18:53:59,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-24 18:53:59,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-24 18:53:59,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-24 18:53:59,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-24 18:53:59,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-24 18:53:59,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-24 18:53:59,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-24 18:53:59,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-24 18:53:59,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-24 18:53:59,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-24 18:53:59,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-24 18:53:59,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-24 18:53:59,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-24 18:53:59,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-24 18:53:59,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-24 18:53:59,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-24 18:53:59,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-24 18:53:59,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-24 18:53:59,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-24 18:53:59,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-24 18:53:59,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-24 18:53:59,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-24 18:53:59,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-24 18:53:59,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-24 18:53:59,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-24 18:53:59,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-24 18:53:59,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-24 18:53:59,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-24 18:53:59,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-24 18:53:59,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-24 18:53:59,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-24 18:53:59,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-24 18:53:59,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-24 18:53:59,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-24 18:53:59,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-24 18:53:59,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-24 18:53:59,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-24 18:53:59,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-24 18:53:59,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-24 18:53:59,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-24 18:53:59,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-24 18:53:59,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-24 18:53:59,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-24 18:53:59,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-24 18:53:59,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-24 18:53:59,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-24 18:53:59,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-24 18:53:59,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-24 18:53:59,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-24 18:53:59,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-24 18:53:59,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-24 18:53:59,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-24 18:53:59,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-24 18:53:59,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-24 18:53:59,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-24 18:53:59,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-24 18:53:59,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-24 18:53:59,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-24 18:53:59,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-24 18:54:00,101 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-24 18:54:00,101 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-24 18:54:00,102 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.014034436254867E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-24 18:54:00,330 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 322 ms to try 5000 different iterations. Found a solution that moves 26 regions; Going from a computed imbalance of 4.014034436254867E-5 to a new imbalance of 6.580041563428285E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0028888888888888888); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-24 18:54:00,330 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1547672429:83 , srv1665816852:83 , srv1808079066:83 , srv185270833:83 , srv1860215688:83 , srv1903764624:83 , srv416147470:83 , srv742617979:83 , srv1023584840:84 , srv1043766822:84 , srv1048342159:84 , srv1074032775:84 , srv1091572319:84 , srv1126581485:84 , srv1139130635:84 , srv1164812150:84 , srv121677622:84 , srv1287323990:84 , srv1307509847:84 , srv1339443792:84 , srv1360669737:84 , srv1367650570:84 , srv1369157131:84 , srv1380370040:84 , srv1381613319:84 , srv1472384535:84 , srv1494471759:84 , srv1587245812:84 , srv1592319761:84 , srv1597122513:84 , srv1597941916:84 , srv1619877781:84 , srv1621927943:84 , srv1649760030:84 , srv1662650376:84 , srv1742044892:84 , srv18261986:84 , srv1866692491:84 , srv1869856841:84 , srv1879841384:84 , srv1908831139:84 , srv1927401991:84 , srv1960961942:84 , srv1973534617:84 , srv1975938425:84 , srv2010564722:84 , srv2055480293:84 , srv2072203922:84 , srv2085304884:84 , srv2108356633:84 , srv2109090951:84 , srv2122891065:84 , srv2146857799:84 , srv215950180:84 , srv2308066:84 , srv244307830:84 , srv259432567:84 , srv281566902:84 , srv347311582:84 , srv360375322:84 , srv397219910:84 , srv405534247:84 , srv414349787:84 , srv415699919:84 , srv422857068:84 , srv442300169:84 , srv489311493:84 , srv498792760:84 , srv508266332:84 , srv571061939:84 , srv57233364:84 , srv575988847:84 , srv585104548:84 , srv586240875:84 , srv626516098:84 , srv690441477:84 , srv724049186:84 , srv734066907:84 , srv734188659:84 , srv762440166:84 , srv767059886:84 , srv770685906:84 , srv837652972:84 , srv839028719:84 , srv848019054:84 , srv85163490:84 , srv908832985:84 , srv909140491:84 , srv958094748:84 , srv964302366:84 , srv979375367:84 , srv1390751201:85 , srv1712099139:85 , srv1798318090:85 , srv1881193288:85 , srv2066327066:85 , srv275909546:85 , srv547030520:85 , srv705680376:85 , srv1561882887:684 } 2023-02-24 18:54:00,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-24 18:54:00,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1381613319=16, srv586240875=80, srv1367650570=13, srv1662650376=29, srv1808079066=34, srv1164812150=7, srv767059886=89, srv734188659=86, srv1649760030=28, srv571061939=76, srv734066907=85, srv405534247=66, srv839028719=92, srv2010564722=48, srv724049186=84, srv705680376=83, srv1091572319=4, srv1619877781=26, srv2085304884=52, srv1927401991=44, srv416147470=69, srv964302366=98, srv414349787=67, srv2146857799=56, srv415699919=68, srv979375367=99, srv1973534617=46, srv275909546=61, srv742617979=87, srv1023584840=0, srv1742044892=32, srv547030520=75, srv585104548=79, srv121677622=8, srv244307830=59, srv690441477=82, srv1665816852=30, srv848019054=93, srv259432567=60, srv397219910=65, srv498792760=73, srv1903764624=42, srv1139130635=6, srv185270833=36, srv1960961942=45, srv1561882887=21, srv2108356633=53, srv2072203922=51, srv489311493=72, srv1048342159=2, srv762440166=88, srv85163490=94, srv1860215688=37, srv2055480293=49, srv1597941916=25, srv958094748=97, srv2308066=58, srv1287323990=9, srv1380370040=15, srv1597122513=24, srv1472384535=18, srv1712099139=31, srv1494471759=19, srv18261986=35, srv442300169=71, srv1869856841=39, srv1360669737=12, srv1126581485=5, srv57233364=77, srv575988847=78, srv770685906=90, srv1043766822=1, srv281566902=62, srv1881193288=41, srv1390751201=17, srv1369157131=14, srv1798318090=33, srv837652972=91, srv1879841384=40, srv2122891065=55, srv2109090951=54, srv1339443792=11, srv422857068=70, srv1621927943=27, srv1908831139=43, srv1074032775=3, srv1592319761=23, srv1866692491=38, srv909140491=96, srv508266332=74, srv360375322=64, srv1307509847=10, srv2066327066=50, srv626516098=81, srv908832985=95, srv215950180=57, srv1975938425=47, srv1547672429=20, srv1587245812=22, srv347311582=63} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-24 18:54:00,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-24 18:54:00,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-24 18:54:00,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-24 18:54:00,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-24 18:54:00,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-24 18:54:00,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-24 18:54:00,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-24 18:54:00,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-24 18:54:00,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-24 18:54:00,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-24 18:54:00,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-24 18:54:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-24 18:54:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-24 18:54:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-24 18:54:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-24 18:54:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-24 18:54:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-24 18:54:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-24 18:54:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-24 18:54:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-24 18:54:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-24 18:54:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-24 18:54:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-24 18:54:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-24 18:54:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-24 18:54:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-24 18:54:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-24 18:54:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-24 18:54:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-24 18:54:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-24 18:54:00,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-24 18:54:00,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-24 18:54:00,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-24 18:54:00,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-24 18:54:00,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-24 18:54:00,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-24 18:54:00,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-24 18:54:00,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-24 18:54:00,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-24 18:54:00,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-24 18:54:00,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-24 18:54:00,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-24 18:54:00,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-24 18:54:00,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-24 18:54:00,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-24 18:54:00,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-24 18:54:00,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-24 18:54:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-24 18:54:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-24 18:54:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-24 18:54:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-24 18:54:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-24 18:54:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-24 18:54:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-24 18:54:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-24 18:54:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-24 18:54:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-24 18:54:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-24 18:54:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-24 18:54:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-24 18:54:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-24 18:54:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-24 18:54:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-24 18:54:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-24 18:54:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-24 18:54:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-24 18:54:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-24 18:54:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-24 18:54:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-24 18:54:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-24 18:54:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-24 18:54:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-24 18:54:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-24 18:54:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-24 18:54:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-24 18:54:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-24 18:54:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-24 18:54:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-24 18:54:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-24 18:54:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-24 18:54:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-24 18:54:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-24 18:54:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-24 18:54:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-24 18:54:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-24 18:54:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-24 18:54:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-24 18:54:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-24 18:54:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-24 18:54:00,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-24 18:54:00,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-24 18:54:00,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-24 18:54:00,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-24 18:54:00,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-24 18:54:00,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-24 18:54:00,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-24 18:54:00,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-24 18:54:00,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-24 18:54:00,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-24 18:54:00,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-24 18:54:00,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-24 18:54:00,459 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.397985672269699E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-24 18:54:00,460 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1547672429:83 , srv1665816852:83 , srv1808079066:83 , srv185270833:83 , srv1860215688:83 , srv1903764624:83 , srv416147470:83 , srv742617979:83 , srv1023584840:84 , srv1043766822:84 , srv1048342159:84 , srv1074032775:84 , srv1091572319:84 , srv1126581485:84 , srv1139130635:84 , srv1164812150:84 , srv121677622:84 , srv1287323990:84 , srv1307509847:84 , srv1339443792:84 , srv1360669737:84 , srv1367650570:84 , srv1369157131:84 , srv1380370040:84 , srv1381613319:84 , srv1472384535:84 , srv1494471759:84 , srv1587245812:84 , srv1592319761:84 , srv1597122513:84 , srv1597941916:84 , srv1619877781:84 , srv1621927943:84 , srv1649760030:84 , srv1662650376:84 , srv1742044892:84 , srv18261986:84 , srv1866692491:84 , srv1869856841:84 , srv1879841384:84 , srv1908831139:84 , srv1927401991:84 , srv1960961942:84 , srv1973534617:84 , srv1975938425:84 , srv2010564722:84 , srv2055480293:84 , srv2072203922:84 , srv2085304884:84 , srv2108356633:84 , srv2109090951:84 , srv2122891065:84 , srv2146857799:84 , srv215950180:84 , srv2308066:84 , srv244307830:84 , srv259432567:84 , srv281566902:84 , srv347311582:84 , srv360375322:84 , srv397219910:84 , srv405534247:84 , srv414349787:84 , srv415699919:84 , srv422857068:84 , srv442300169:84 , srv489311493:84 , srv498792760:84 , srv508266332:84 , srv571061939:84 , srv57233364:84 , srv575988847:84 , srv585104548:84 , srv586240875:84 , srv626516098:84 , srv690441477:84 , srv724049186:84 , srv734066907:84 , srv734188659:84 , srv762440166:84 , srv767059886:84 , srv770685906:84 , srv837652972:84 , srv839028719:84 , srv848019054:84 , srv85163490:84 , srv908832985:84 , srv909140491:84 , srv958094748:84 , srv964302366:84 , srv979375367:84 , srv1390751201:85 , srv1712099139:85 , srv1798318090:85 , srv1881193288:85 , srv2066327066:85 , srv275909546:85 , srv547030520:85 , srv705680376:85 , srv1561882887:684 } 2023-02-24 18:54:00,486 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=368 (was 347) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=5974 (was 6559) 2023-02-24 18:54:00,494 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=368, ProcessCount=171, AvailableMemoryMB=5974 2023-02-24 18:54:00,495 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-24 18:54:00,495 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-02-24 18:54:00,507 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1138855365:84 , srv116476001:84 , srv1246540921:84 , srv1246787876:84 , srv1258543504:84 , srv1357611899:84 , srv1358629512:84 , srv1384206021:84 , srv1403808227:84 , srv1430931913:84 , srv1440988313:84 , srv1531107324:84 , srv1643479851:84 , srv1663033694:84 , srv1687175424:84 , srv189875352:84 , srv1986011933:84 , srv2000035342:84 , srv244874367:84 , srv354353254:84 , srv368383835:84 , srv436348818:84 , srv503435151:84 , srv770015952:84 , srv775514348:84 , srv780343051:84 , srv814050481:84 , srv817455284:84 , srv99306563:84 , srv350405457:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-02-24 18:54:00,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-24 18:54:00,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1246540921=2, srv1643479851=12, srv1138855365=0, srv770015952=24, srv1531107324=11, srv1430931913=9, srv1663033694=13, srv2000035342=17, srv244874367=18, srv354353254=20, srv99306563=29, srv1986011933=16, srv350405457=19, srv436348818=22, srv189875352=15, srv1246787876=3, srv1440988313=10, srv116476001=1, srv1384206021=7, srv775514348=25, srv817455284=28, srv780343051=26, srv1357611899=5, srv1258543504=4, srv1403808227=8, srv1687175424=14, srv814050481=27, srv1358629512=6, srv368383835=21, srv503435151=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-24 18:54:00,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-24 18:54:00,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-24 18:54:00,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-24 18:54:00,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-24 18:54:00,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-24 18:54:00,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-24 18:54:00,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-24 18:54:00,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-24 18:54:00,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-24 18:54:00,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-24 18:54:00,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-24 18:54:00,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-24 18:54:00,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-24 18:54:00,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-24 18:54:00,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-24 18:54:00,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-24 18:54:00,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-24 18:54:00,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-24 18:54:00,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-24 18:54:00,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-24 18:54:00,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-24 18:54:00,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-24 18:54:00,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-24 18:54:00,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-24 18:54:00,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-24 18:54:00,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-24 18:54:00,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-24 18:54:00,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-24 18:54:00,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-24 18:54:00,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-24 18:54:00,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-24 18:54:00,535 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1138131115328556, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-24 18:54:00,536 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.991255166466238, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1138131115328556, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-02-24 18:56:00,523 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6218266 different iterations. Found a solution that moves 1866 regions; Going from a computed imbalance of 0.991255166466238 to a new imbalance of 4.3553370884861656E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6911111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-24 18:56:00,531 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1138855365:90 , srv116476001:90 , srv1246540921:90 , srv1246787876:90 , srv1258543504:90 , srv1357611899:90 , srv1358629512:90 , srv1384206021:90 , srv1403808227:90 , srv1430931913:90 , srv1440988313:90 , srv1531107324:90 , srv1643479851:90 , srv1663033694:90 , srv1687175424:90 , srv189875352:90 , srv1986011933:90 , srv2000035342:90 , srv244874367:90 , srv350405457:90 , srv354353254:90 , srv368383835:90 , srv436348818:90 , srv503435151:90 , srv770015952:90 , srv775514348:90 , srv780343051:90 , srv814050481:90 , srv817455284:90 , srv99306563:90 } 2023-02-24 18:56:00,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-24 18:56:00,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1246540921=2, srv1643479851=12, srv1138855365=0, srv770015952=24, srv1531107324=11, srv1430931913=9, srv1663033694=13, srv2000035342=17, srv244874367=18, srv354353254=20, srv99306563=29, srv1986011933=16, srv350405457=19, srv436348818=22, srv189875352=15, srv1246787876=3, srv1440988313=10, srv116476001=1, srv1384206021=7, srv775514348=25, srv817455284=28, srv780343051=26, srv1357611899=5, srv1258543504=4, srv1403808227=8, srv1687175424=14, srv814050481=27, srv1358629512=6, srv368383835=21, srv503435151=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-24 18:56:00,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-24 18:56:00,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-24 18:56:00,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-24 18:56:00,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-24 18:56:00,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-24 18:56:00,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-24 18:56:00,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-24 18:56:00,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-24 18:56:00,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-24 18:56:00,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-24 18:56:00,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-24 18:56:00,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-24 18:56:00,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-24 18:56:00,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-24 18:56:00,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-24 18:56:00,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-24 18:56:00,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-24 18:56:00,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-24 18:56:00,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-24 18:56:00,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-24 18:56:00,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-24 18:56:00,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-24 18:56:00,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-24 18:56:00,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-24 18:56:00,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-24 18:56:00,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-24 18:56:00,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-24 18:56:00,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-24 18:56:00,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-24 18:56:00,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-24 18:56:00,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-24 18:56:00,548 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-24 18:56:00,549 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1138855365:90 , srv116476001:90 , srv1246540921:90 , srv1246787876:90 , srv1258543504:90 , srv1357611899:90 , srv1358629512:90 , srv1384206021:90 , srv1403808227:90 , srv1430931913:90 , srv1440988313:90 , srv1531107324:90 , srv1643479851:90 , srv1663033694:90 , srv1687175424:90 , srv189875352:90 , srv1986011933:90 , srv2000035342:90 , srv244874367:90 , srv350405457:90 , srv354353254:90 , srv368383835:90 , srv436348818:90 , srv503435151:90 , srv770015952:90 , srv775514348:90 , srv780343051:90 , srv814050481:90 , srv817455284:90 , srv99306563:90 } 2023-02-24 18:56:00,561 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=213 (was 368), ProcessCount=173 (was 171) - ProcessCount LEAK? -, AvailableMemoryMB=5951 (was 5974)