2023-12-30 15:53:14,238 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-12-30 15:53:14,566 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-12-30 15:53:14,678 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-12-30 15:53:14,746 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-30 15:53:14,756 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-30 15:53:14,770 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=285, ProcessCount=171, AvailableMemoryMB=5106 2023-12-30 15:53:14,772 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-30 15:53:14,772 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-30 15:53:14,911 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1008561695:84 , srv1017677068:84 , srv10364307:84 , srv1038319955:84 , srv1065556504:84 , srv109389693:84 , srv1104564957:84 , srv1123674319:84 , srv1166122104:84 , srv1195151220:84 , srv122076318:84 , srv1226306891:84 , srv1246178217:84 , srv1254416310:84 , srv1278496520:84 , srv1279230135:84 , srv1287349010:84 , srv1320898657:84 , srv1351508554:84 , srv1387357953:84 , srv1387992390:84 , srv1421282831:84 , srv1421992201:84 , srv143121918:84 , srv1476788133:84 , srv1498816146:84 , srv1569821429:84 , srv1586398771:84 , srv1602833799:84 , srv1646347881:84 , srv1664905839:84 , srv1725237300:84 , srv1729560149:84 , srv1746422084:84 , srv1804208178:84 , srv1807004428:84 , srv1823529363:84 , srv1823872841:84 , srv184464993:84 , srv1898865425:84 , srv1915870072:84 , srv1934249993:84 , srv1945048597:84 , srv1959361613:84 , srv1999603125:84 , srv20035614:84 , srv200384619:84 , srv2006546142:84 , srv2009390650:84 , srv2071995786:84 , srv2135476058:84 , srv2137846315:84 , srv286624428:84 , srv299790650:84 , srv313556931:84 , srv322238762:84 , srv325131401:84 , srv333262429:84 , srv350525406:84 , srv377061447:84 , srv409644759:84 , srv438021544:84 , srv496552156:84 , srv559896043:84 , srv575847746:84 , srv578066910:84 , srv582462575:84 , srv623150267:84 , srv645070831:84 , srv668415034:84 , srv672390567:84 , srv673463037:84 , srv687796086:84 , srv696839131:84 , srv696869375:84 , srv714457966:84 , srv740963470:84 , srv74807007:84 , srv759760060:84 , srv762469662:84 , srv764814827:84 , srv779599341:84 , srv781404718:84 , srv787366646:84 , srv788358914:84 , srv797824707:84 , srv806055536:84 , srv807290920:84 , srv825228751:84 , srv887838442:84 , srv897834965:84 , srv912439168:84 , srv913165760:84 , srv926238398:84 , srv928403914:84 , srv932584258:84 , srv945521501:84 , srv948355809:84 , srv996204978:84 , srv1723046767:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-12-30 15:53:14,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-30 15:53:14,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv787366646=84, srv806055536=87, srv926238398=94, srv807290920=88, srv578066910=66, srv673463037=72, srv1945048597=43, srv2135476058=51, srv2009390650=49, srv948355809=98, srv122076318=10, srv377061447=60, srv1387992390=20, srv1498816146=25, srv1421282831=21, srv1476788133=24, srv672390567=71, srv759760060=79, srv10364307=2, srv913165760=93, srv932584258=96, srv1729560149=33, srv2006546142=48, srv912439168=92, srv313556931=55, srv1166122104=8, srv438021544=62, srv1279230135=15, srv1278496520=14, srv1195151220=9, srv781404718=83, srv740963470=77, srv687796086=73, srv1569821429=26, srv1038319955=3, srv1959361613=44, srv1804208178=35, srv20035614=46, srv350525406=59, srv1823872841=38, srv945521501=97, srv1807004428=36, srv322238762=56, srv1421992201=22, srv1602833799=28, srv200384619=47, srv1664905839=30, srv623150267=68, srv299790650=54, srv1104564957=6, srv764814827=81, srv109389693=5, srv825228751=89, srv409644759=61, srv696839131=74, srv887838442=90, srv1017677068=1, srv2071995786=50, srv762469662=80, srv1746422084=34, srv645070831=69, srv1065556504=4, srv184464993=39, srv1351508554=18, srv996204978=99, srv1123674319=7, srv559896043=64, srv1387357953=19, srv1999603125=45, srv286624428=53, srv496552156=63, srv696869375=75, srv797824707=86, srv779599341=82, srv897834965=91, srv788358914=85, srv1320898657=17, srv74807007=78, srv1915870072=41, srv668415034=70, srv1226306891=11, srv582462575=67, srv1898865425=40, srv1008561695=0, srv2137846315=52, srv1823529363=37, srv575847746=65, srv143121918=23, srv928403914=95, srv1254416310=13, srv1586398771=27, srv1287349010=16, srv1934249993=42, srv325131401=57, srv1646347881=29, srv333262429=58, srv714457966=76, srv1725237300=32, srv1246178217=12, srv1723046767=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-30 15:53:15,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-30 15:53:15,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-30 15:53:15,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-30 15:53:15,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-30 15:53:15,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-30 15:53:15,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-30 15:53:15,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-30 15:53:15,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-30 15:53:15,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-30 15:53:15,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-30 15:53:15,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-30 15:53:15,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-30 15:53:15,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-30 15:53:15,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-30 15:53:15,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-30 15:53:15,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-30 15:53:15,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-30 15:53:15,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-30 15:53:15,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-30 15:53:15,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-30 15:53:15,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-30 15:53:15,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-30 15:53:15,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-30 15:53:15,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-30 15:53:15,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-30 15:53:15,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-30 15:53:15,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-30 15:53:15,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-30 15:53:15,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-30 15:53:15,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-30 15:53:15,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-30 15:53:15,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-30 15:53:15,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-30 15:53:15,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-30 15:53:15,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-30 15:53:15,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-30 15:53:15,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-30 15:53:15,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-30 15:53:15,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-30 15:53:15,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-30 15:53:15,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-30 15:53:15,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-30 15:53:15,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-30 15:53:15,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-30 15:53:15,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-30 15:53:15,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-30 15:53:15,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-30 15:53:15,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-30 15:53:15,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-30 15:53:15,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-30 15:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-30 15:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-30 15:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-30 15:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-30 15:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-30 15:53:15,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-30 15:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-30 15:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-30 15:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-30 15:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-30 15:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-30 15:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-30 15:53:15,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-30 15:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-30 15:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-30 15:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-30 15:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-30 15:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-30 15:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-30 15:53:15,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-30 15:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-30 15:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-30 15:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-30 15:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-30 15:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-30 15:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-30 15:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-30 15:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-30 15:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-30 15:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-30 15:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-30 15:53:15,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-30 15:53:15,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-30 15:53:15,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-30 15:53:15,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-30 15:53:15,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-30 15:53:15,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-30 15:53:15,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-30 15:53:15,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-30 15:53:15,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-30 15:53:15,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-30 15:53:15,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-30 15:53:15,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-30 15:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-30 15:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-30 15:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-30 15:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-30 15:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-30 15:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-30 15:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-30 15:53:15,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-30 15:53:15,233 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-30 15:53:15,233 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-30 15:53:15,234 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-30 15:53:15,660 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 580 ms to try 5000 different iterations. Found a solution that moves 4638 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18447443787526374. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037512104709687945, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05604831970196913, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5153333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.037512104709687945, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1795, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24891666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-30 15:53:15,678 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1823529363:80 , srv1387357953:81 , srv645070831:81 , srv1746422084:82 , srv438021544:82 , srv807290920:82 , srv913165760:82 , srv1287349010:83 , srv322238762:83 , srv333262429:83 , srv788358914:83 , srv1017677068:84 , srv1038319955:84 , srv143121918:84 , srv1476788133:84 , srv1646347881:84 , srv1729560149:84 , srv582462575:84 , srv668415034:84 , srv673463037:84 , srv714457966:84 , srv764814827:84 , srv825228751:84 , srv945521501:84 , srv10364307:85 , srv1166122104:85 , srv1195151220:85 , srv1278496520:85 , srv1421282831:85 , srv1664905839:85 , srv2071995786:85 , srv696869375:85 , srv797824707:85 , srv806055536:85 , srv887838442:85 , srv926238398:85 , srv928403914:85 , srv996204978:85 , srv1008561695:86 , srv1569821429:86 , srv1804208178:86 , srv1807004428:86 , srv1823872841:86 , srv1934249993:86 , srv200384619:86 , srv2006546142:86 , srv313556931:86 , srv409644759:86 , srv687796086:86 , srv897834965:86 , srv122076318:87 , srv1246178217:87 , srv1279230135:87 , srv1725237300:87 , srv184464993:87 , srv1945048597:87 , srv20035614:87 , srv377061447:87 , srv559896043:87 , srv672390567:87 , srv779599341:87 , srv948355809:87 , srv1254416310:88 , srv1498816146:88 , srv1898865425:88 , srv1915870072:88 , srv1959361613:88 , srv1999603125:88 , srv2009390650:88 , srv286624428:88 , srv578066910:88 , srv74807007:88 , srv762469662:88 , srv912439168:88 , srv1123674319:89 , srv1226306891:89 , srv1421992201:89 , srv325131401:89 , srv350525406:89 , srv575847746:89 , srv696839131:89 , srv740963470:89 , srv759760060:89 , srv1320898657:90 , srv1351508554:90 , srv1602833799:90 , srv2135476058:90 , srv2137846315:90 , srv787366646:90 , srv932584258:90 , srv109389693:91 , srv623150267:91 , srv1065556504:92 , srv1104564957:92 , srv1387992390:92 , srv1586398771:92 , srv496552156:92 , srv781404718:92 , srv299790650:96 , srv1723046767:423 } 2023-12-30 15:53:15,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-30 15:53:15,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv787366646=84, srv806055536=87, srv926238398=94, srv807290920=88, srv578066910=66, srv673463037=72, srv1945048597=43, srv2135476058=51, srv2009390650=49, srv948355809=98, srv122076318=10, srv377061447=60, srv1387992390=20, srv1498816146=25, srv1421282831=21, srv1476788133=24, srv672390567=71, srv759760060=79, srv10364307=2, srv913165760=93, srv932584258=96, srv1729560149=33, srv2006546142=48, srv912439168=92, srv313556931=55, srv1166122104=8, srv438021544=62, srv1279230135=15, srv1278496520=14, srv1195151220=9, srv781404718=83, srv740963470=77, srv687796086=73, srv1569821429=26, srv1038319955=3, srv1959361613=44, srv1804208178=35, srv20035614=46, srv350525406=59, srv1823872841=38, srv945521501=97, srv1807004428=36, srv322238762=56, srv1421992201=22, srv1602833799=28, srv200384619=47, srv1664905839=30, srv623150267=68, srv299790650=54, srv1104564957=6, srv764814827=81, srv109389693=5, srv825228751=89, srv409644759=61, srv696839131=74, srv887838442=90, srv1017677068=1, srv2071995786=50, srv762469662=80, srv1746422084=34, srv645070831=69, srv1065556504=4, srv184464993=39, srv1351508554=18, srv996204978=99, srv1123674319=7, srv559896043=64, srv1387357953=19, srv1999603125=45, srv286624428=53, srv496552156=63, srv696869375=75, srv797824707=86, srv779599341=82, srv897834965=91, srv788358914=85, srv1320898657=17, srv74807007=78, srv1915870072=41, srv668415034=70, srv1226306891=11, srv582462575=67, srv1898865425=40, srv1008561695=0, srv2137846315=52, srv1823529363=37, srv575847746=65, srv143121918=23, srv928403914=95, srv1254416310=13, srv1586398771=27, srv1287349010=16, srv1934249993=42, srv325131401=57, srv1646347881=29, srv333262429=58, srv714457966=76, srv1725237300=32, srv1246178217=12, srv1723046767=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-30 15:53:15,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-30 15:53:15,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-30 15:53:15,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-30 15:53:15,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-30 15:53:15,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-30 15:53:15,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-30 15:53:15,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-30 15:53:15,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-30 15:53:15,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-30 15:53:15,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-30 15:53:15,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-30 15:53:15,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-30 15:53:15,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-30 15:53:15,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-30 15:53:15,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-30 15:53:15,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-30 15:53:15,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-30 15:53:15,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-30 15:53:15,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-30 15:53:15,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-30 15:53:15,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-30 15:53:15,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-30 15:53:15,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-30 15:53:15,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-30 15:53:15,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-30 15:53:15,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-30 15:53:15,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-30 15:53:15,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-30 15:53:15,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-30 15:53:15,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-30 15:53:15,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-30 15:53:15,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-30 15:53:15,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-30 15:53:15,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-30 15:53:15,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-30 15:53:15,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-30 15:53:15,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-30 15:53:15,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-30 15:53:15,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-30 15:53:15,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-30 15:53:15,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-30 15:53:15,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-30 15:53:15,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-30 15:53:15,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-30 15:53:15,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-30 15:53:15,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-30 15:53:15,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-30 15:53:15,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-30 15:53:15,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-30 15:53:15,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-30 15:53:15,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-30 15:53:15,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-30 15:53:15,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-30 15:53:15,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-30 15:53:15,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-30 15:53:15,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-30 15:53:15,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-30 15:53:15,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-30 15:53:15,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-30 15:53:15,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-30 15:53:15,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-30 15:53:15,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-30 15:53:15,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-30 15:53:15,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-30 15:53:15,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-30 15:53:15,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-30 15:53:15,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-30 15:53:15,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-30 15:53:15,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-30 15:53:15,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-30 15:53:15,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-30 15:53:15,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-30 15:53:15,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-30 15:53:15,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-30 15:53:15,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-30 15:53:15,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-30 15:53:15,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-30 15:53:15,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-30 15:53:15,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-30 15:53:15,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-30 15:53:15,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-30 15:53:15,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-30 15:53:15,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-30 15:53:15,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-30 15:53:15,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-30 15:53:15,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-30 15:53:15,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-30 15:53:15,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-30 15:53:15,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-30 15:53:15,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-30 15:53:15,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-30 15:53:15,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-30 15:53:15,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-30 15:53:15,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-30 15:53:15,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-30 15:53:15,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-30 15:53:15,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-30 15:53:15,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-30 15:53:15,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-30 15:53:15,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-30 15:53:15,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-30 15:53:15,844 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037512104709687945, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05604831970196913, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.037512104709687945, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1795, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24891666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-30 15:53:15,845 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-30 15:53:15,845 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18444196190514092, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037512104709687945, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05604831970196913, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.037512104709687945, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1795, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24891666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-30 15:53:16,152 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 424 ms to try 5000 different iterations. Found a solution that moves 2751 regions; Going from a computed imbalance of 0.18444196190514092 to a new imbalance of 0.01619598093090444. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02198243241952962); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03353474503524851, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.30566666666666664, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02198243241952962); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04683333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-30 15:53:16,162 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv299790650:75 , srv1586398771:79 , srv1065556504:80 , srv1387992390:80 , srv496552156:80 , srv109389693:81 , srv2135476058:81 , srv578066910:81 , srv759760060:81 , srv781404718:81 , srv1123674319:82 , srv1226306891:82 , srv1351508554:82 , srv2137846315:82 , srv623150267:82 , srv696839131:82 , srv932584258:82 , srv122076318:83 , srv1279230135:83 , srv1498816146:83 , srv1602833799:83 , srv1959361613:83 , srv350525406:83 , srv575847746:83 , srv74807007:83 , srv762469662:83 , srv787366646:83 , srv10364307:84 , srv1246178217:84 , srv1320898657:84 , srv1725237300:84 , srv184464993:84 , srv1999603125:84 , srv2009390650:84 , srv377061447:84 , srv740963470:84 , srv1195151220:85 , srv1421992201:85 , srv1664905839:85 , srv1807004428:85 , srv1898865425:85 , srv1945048597:85 , srv200384619:85 , srv286624428:85 , srv313556931:85 , srv325131401:85 , srv409644759:85 , srv559896043:85 , srv672390567:85 , srv897834965:85 , srv912439168:85 , srv1038319955:86 , srv1254416310:86 , srv1421282831:86 , srv1569821429:86 , srv1915870072:86 , srv20035614:86 , srv2071995786:86 , srv687796086:86 , srv696869375:86 , srv825228751:86 , srv887838442:86 , srv928403914:86 , srv996204978:86 , srv1287349010:87 , srv143121918:87 , srv1476788133:87 , srv1646347881:87 , srv1804208178:87 , srv1823872841:87 , srv582462575:87 , srv673463037:87 , srv764814827:87 , srv779599341:87 , srv797824707:87 , srv926238398:87 , srv948355809:87 , srv1017677068:88 , srv1166122104:88 , srv2006546142:88 , srv438021544:88 , srv714457966:88 , srv788358914:88 , srv1387357953:89 , srv1934249993:89 , srv322238762:89 , srv668415034:89 , srv806055536:89 , srv807290920:89 , srv1008561695:90 , srv1278496520:90 , srv1746422084:90 , srv333262429:90 , srv913165760:90 , srv945521501:90 , srv1104564957:92 , srv1729560149:92 , srv645070831:92 , srv1823529363:93 , srv1723046767:546 } 2023-12-30 15:53:16,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-30 15:53:16,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv787366646=84, srv806055536=87, srv926238398=94, srv807290920=88, srv578066910=66, srv673463037=72, srv1945048597=43, srv2135476058=51, srv2009390650=49, srv948355809=98, srv122076318=10, srv377061447=60, srv1387992390=20, srv1498816146=25, srv1421282831=21, srv1476788133=24, srv672390567=71, srv759760060=79, srv10364307=2, srv913165760=93, srv932584258=96, srv1729560149=33, srv2006546142=48, srv912439168=92, srv313556931=55, srv1166122104=8, srv438021544=62, srv1279230135=15, srv1278496520=14, srv1195151220=9, srv781404718=83, srv740963470=77, srv687796086=73, srv1569821429=26, srv1038319955=3, srv1959361613=44, srv1804208178=35, srv20035614=46, srv350525406=59, srv1823872841=38, srv945521501=97, srv1807004428=36, srv322238762=56, srv1421992201=22, srv1602833799=28, srv200384619=47, srv1664905839=30, srv623150267=68, srv299790650=54, srv1104564957=6, srv764814827=81, srv109389693=5, srv825228751=89, srv409644759=61, srv696839131=74, srv887838442=90, srv1017677068=1, srv2071995786=50, srv762469662=80, srv1746422084=34, srv645070831=69, srv1065556504=4, srv184464993=39, srv1351508554=18, srv996204978=99, srv1123674319=7, srv559896043=64, srv1387357953=19, srv1999603125=45, srv286624428=53, srv496552156=63, srv696869375=75, srv797824707=86, srv779599341=82, srv897834965=91, srv788358914=85, srv1320898657=17, srv74807007=78, srv1915870072=41, srv668415034=70, srv1226306891=11, srv582462575=67, srv1898865425=40, srv1008561695=0, srv2137846315=52, srv1823529363=37, srv575847746=65, srv143121918=23, srv928403914=95, srv1254416310=13, srv1586398771=27, srv1287349010=16, srv1934249993=42, srv325131401=57, srv1646347881=29, srv333262429=58, srv714457966=76, srv1725237300=32, srv1246178217=12, srv1723046767=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-30 15:53:16,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-30 15:53:16,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-30 15:53:16,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-30 15:53:16,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-30 15:53:16,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-30 15:53:16,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-30 15:53:16,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-30 15:53:16,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-30 15:53:16,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-30 15:53:16,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-30 15:53:16,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-30 15:53:16,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-30 15:53:16,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-30 15:53:16,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-30 15:53:16,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-30 15:53:16,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-30 15:53:16,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-30 15:53:16,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-30 15:53:16,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-30 15:53:16,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-30 15:53:16,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-30 15:53:16,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-30 15:53:16,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-30 15:53:16,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-30 15:53:16,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-30 15:53:16,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-30 15:53:16,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-30 15:53:16,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-30 15:53:16,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-30 15:53:16,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-30 15:53:16,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-30 15:53:16,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-30 15:53:16,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-30 15:53:16,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-30 15:53:16,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-30 15:53:16,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-30 15:53:16,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-30 15:53:16,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-30 15:53:16,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-30 15:53:16,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-30 15:53:16,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-30 15:53:16,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-30 15:53:16,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-30 15:53:16,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-30 15:53:16,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-30 15:53:16,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-30 15:53:16,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-30 15:53:16,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-30 15:53:16,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-30 15:53:16,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-30 15:53:16,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-30 15:53:16,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-30 15:53:16,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-30 15:53:16,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-30 15:53:16,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-30 15:53:16,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-30 15:53:16,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-30 15:53:16,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-30 15:53:16,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-30 15:53:16,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-30 15:53:16,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-30 15:53:16,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-30 15:53:16,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-30 15:53:16,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-30 15:53:16,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-30 15:53:16,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-30 15:53:16,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-30 15:53:16,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-30 15:53:16,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-30 15:53:16,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-30 15:53:16,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-30 15:53:16,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-30 15:53:16,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-30 15:53:16,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-30 15:53:16,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-30 15:53:16,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-30 15:53:16,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-30 15:53:16,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-30 15:53:16,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-30 15:53:16,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-30 15:53:16,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-30 15:53:16,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-30 15:53:16,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-30 15:53:16,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-30 15:53:16,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-30 15:53:16,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-30 15:53:16,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-30 15:53:16,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-30 15:53:16,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-30 15:53:16,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-30 15:53:16,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-30 15:53:16,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-30 15:53:16,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-30 15:53:16,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-30 15:53:16,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-30 15:53:16,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-30 15:53:16,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-30 15:53:16,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-30 15:53:16,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-30 15:53:16,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-30 15:53:16,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-30 15:53:16,312 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02198243241952962); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03353474503524851, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02198243241952962); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04683333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-30 15:53:16,312 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-30 15:53:16,312 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01617671801718993, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02198243241952962); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03353474503524851, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02198243241952962); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04683333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-30 15:53:16,666 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 466 ms to try 5000 different iterations. Found a solution that moves 1078 regions; Going from a computed imbalance of 0.01617671801718993 to a new imbalance of 6.42867503377246E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005311120972618093); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007122465682345129); MoveCostFunction : (multiplier=7.0, imbalance=0.11977777777777777, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005311120972618093); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00475); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-30 15:53:16,670 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1104564957:74 , srv1008561695:81 , srv1254416310:83 , srv1278496520:83 , srv1729560149:83 , srv1934249993:83 , srv325131401:83 , srv806055536:83 , srv948355809:83 , srv1166122104:84 , srv1320898657:84 , srv1421992201:84 , srv1823872841:84 , srv1915870072:84 , srv20035614:84 , srv2006546142:84 , srv286624428:84 , srv333262429:84 , srv623150267:84 , srv668415034:84 , srv740963470:84 , srv779599341:84 , srv781404718:84 , srv945521501:84 , srv1065556504:85 , srv1387992390:85 , srv1804208178:85 , srv1823529363:85 , srv1898865425:85 , srv2009390650:85 , srv2137846315:85 , srv322238762:85 , srv575847746:85 , srv645070831:85 , srv714457966:85 , srv787366646:85 , srv912439168:85 , srv913165760:85 , srv1017677068:86 , srv109389693:86 , srv1123674319:86 , srv1351508554:86 , srv143121918:86 , srv1569821429:86 , srv1602833799:86 , srv1646347881:86 , srv1725237300:86 , srv1746422084:86 , srv1807004428:86 , srv184464993:86 , srv1945048597:86 , srv1999603125:86 , srv200384619:86 , srv2071995786:86 , srv313556931:86 , srv350525406:86 , srv409644759:86 , srv496552156:86 , srv559896043:86 , srv582462575:86 , srv672390567:86 , srv673463037:86 , srv687796086:86 , srv696839131:86 , srv696869375:86 , srv764814827:86 , srv788358914:86 , srv797824707:86 , srv897834965:86 , srv926238398:86 , srv928403914:86 , srv932584258:86 , srv996204978:86 , srv1038319955:87 , srv1195151220:87 , srv122076318:87 , srv1226306891:87 , srv1246178217:87 , srv1287349010:87 , srv1387357953:87 , srv1421282831:87 , srv1476788133:87 , srv1498816146:87 , srv1586398771:87 , srv1664905839:87 , srv1959361613:87 , srv2135476058:87 , srv299790650:87 , srv377061447:87 , srv438021544:87 , srv74807007:87 , srv759760060:87 , srv762469662:87 , srv807290920:87 , srv825228751:87 , srv887838442:87 , srv1279230135:88 , srv578066910:89 , srv10364307:90 , srv1723046767:536 } 2023-12-30 15:53:16,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-30 15:53:16,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv787366646=84, srv806055536=87, srv926238398=94, srv807290920=88, srv578066910=66, srv673463037=72, srv1945048597=43, srv2135476058=51, srv2009390650=49, srv948355809=98, srv122076318=10, srv377061447=60, srv1387992390=20, srv1498816146=25, srv1421282831=21, srv1476788133=24, srv672390567=71, srv759760060=79, srv10364307=2, srv913165760=93, srv932584258=96, srv1729560149=33, srv2006546142=48, srv912439168=92, srv313556931=55, srv1166122104=8, srv438021544=62, srv1279230135=15, srv1278496520=14, srv1195151220=9, srv781404718=83, srv740963470=77, srv687796086=73, srv1569821429=26, srv1038319955=3, srv1959361613=44, srv1804208178=35, srv20035614=46, srv350525406=59, srv1823872841=38, srv945521501=97, srv1807004428=36, srv322238762=56, srv1421992201=22, srv1602833799=28, srv200384619=47, srv1664905839=30, srv623150267=68, srv299790650=54, srv1104564957=6, srv764814827=81, srv109389693=5, srv825228751=89, srv409644759=61, srv696839131=74, srv887838442=90, srv1017677068=1, srv2071995786=50, srv762469662=80, srv1746422084=34, srv645070831=69, srv1065556504=4, srv184464993=39, srv1351508554=18, srv996204978=99, srv1123674319=7, srv559896043=64, srv1387357953=19, srv1999603125=45, srv286624428=53, srv496552156=63, srv696869375=75, srv797824707=86, srv779599341=82, srv897834965=91, srv788358914=85, srv1320898657=17, srv74807007=78, srv1915870072=41, srv668415034=70, srv1226306891=11, srv582462575=67, srv1898865425=40, srv1008561695=0, srv2137846315=52, srv1823529363=37, srv575847746=65, srv143121918=23, srv928403914=95, srv1254416310=13, srv1586398771=27, srv1287349010=16, srv1934249993=42, srv325131401=57, srv1646347881=29, srv333262429=58, srv714457966=76, srv1725237300=32, srv1246178217=12, srv1723046767=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-30 15:53:16,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-30 15:53:16,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-30 15:53:16,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-30 15:53:16,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-30 15:53:16,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-30 15:53:16,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-30 15:53:16,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-30 15:53:16,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-30 15:53:16,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-30 15:53:16,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-30 15:53:16,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-30 15:53:16,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-30 15:53:16,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-30 15:53:16,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-30 15:53:16,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-30 15:53:16,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-30 15:53:16,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-30 15:53:16,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-30 15:53:16,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-30 15:53:16,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-30 15:53:16,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-30 15:53:16,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-30 15:53:16,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-30 15:53:16,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-30 15:53:16,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-30 15:53:16,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-30 15:53:16,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-30 15:53:16,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-30 15:53:16,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-30 15:53:16,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-30 15:53:16,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-30 15:53:16,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-30 15:53:16,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-30 15:53:16,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-30 15:53:16,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-30 15:53:16,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-30 15:53:16,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-30 15:53:16,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-30 15:53:16,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-30 15:53:16,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-30 15:53:16,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-30 15:53:16,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-30 15:53:16,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-30 15:53:16,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-30 15:53:16,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-30 15:53:16,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-30 15:53:16,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-30 15:53:16,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-30 15:53:16,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-30 15:53:16,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-30 15:53:16,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-30 15:53:16,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-30 15:53:16,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-30 15:53:16,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-30 15:53:16,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-30 15:53:16,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-30 15:53:16,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-30 15:53:16,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-30 15:53:16,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-30 15:53:16,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-30 15:53:16,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-30 15:53:16,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-30 15:53:16,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-30 15:53:16,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-30 15:53:16,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-30 15:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-30 15:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-30 15:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-30 15:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-30 15:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-30 15:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-30 15:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-30 15:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-30 15:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-30 15:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-30 15:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-30 15:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-30 15:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-30 15:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-30 15:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-30 15:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-30 15:53:16,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-30 15:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-30 15:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-30 15:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-30 15:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-30 15:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-30 15:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-30 15:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-30 15:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-30 15:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-30 15:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-30 15:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-30 15:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-30 15:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-30 15:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-30 15:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-30 15:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-30 15:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-30 15:53:16,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-30 15:53:16,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-30 15:53:16,807 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005311120972618093); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007122465682345129); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005311120972618093); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00475); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-30 15:53:16,807 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-30 15:53:16,807 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.353191860438246E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005311120972618093); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007122465682345129); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005311120972618093); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00475); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-30 15:53:17,041 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 335 ms to try 5000 different iterations. Found a solution that moves 248 regions; Going from a computed imbalance of 6.353191860438246E-4 to a new imbalance of 2.8071205862537736E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.027555555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-30 15:53:17,042 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv10364307:83 , srv1586398771:83 , srv1602833799:83 , srv1729560149:83 , srv1804208178:83 , srv350525406:83 , srv1017677068:84 , srv109389693:84 , srv1104564957:84 , srv1123674319:84 , srv1226306891:84 , srv1246178217:84 , srv1254416310:84 , srv1278496520:84 , srv1279230135:84 , srv1351508554:84 , srv1387357953:84 , srv1421282831:84 , srv143121918:84 , srv1476788133:84 , srv1498816146:84 , srv1569821429:84 , srv1823529363:84 , srv1898865425:84 , srv1915870072:84 , srv1934249993:84 , srv1945048597:84 , srv1959361613:84 , srv1999603125:84 , srv200384619:84 , srv2006546142:84 , srv2135476058:84 , srv299790650:84 , srv377061447:84 , srv496552156:84 , srv559896043:84 , srv578066910:84 , srv645070831:84 , srv672390567:84 , srv687796086:84 , srv74807007:84 , srv762469662:84 , srv779599341:84 , srv797824707:84 , srv806055536:84 , srv887838442:84 , srv912439168:84 , srv926238398:84 , srv932584258:84 , srv945521501:84 , srv1008561695:85 , srv1038319955:85 , srv1065556504:85 , srv1166122104:85 , srv1195151220:85 , srv122076318:85 , srv1287349010:85 , srv1320898657:85 , srv1387992390:85 , srv1421992201:85 , srv1664905839:85 , srv1725237300:85 , srv1746422084:85 , srv1807004428:85 , srv1823872841:85 , srv184464993:85 , srv20035614:85 , srv2009390650:85 , srv2071995786:85 , srv2137846315:85 , srv286624428:85 , srv313556931:85 , srv322238762:85 , srv325131401:85 , srv333262429:85 , srv409644759:85 , srv438021544:85 , srv575847746:85 , srv582462575:85 , srv623150267:85 , srv673463037:85 , srv696839131:85 , srv696869375:85 , srv740963470:85 , srv759760060:85 , srv781404718:85 , srv787366646:85 , srv788358914:85 , srv807290920:85 , srv825228751:85 , srv897834965:85 , srv913165760:85 , srv928403914:85 , srv948355809:85 , srv996204978:85 , srv1646347881:86 , srv668415034:86 , srv714457966:86 , srv764814827:86 , srv1723046767:637 } 2023-12-30 15:53:17,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-30 15:53:17,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv787366646=84, srv806055536=87, srv926238398=94, srv807290920=88, srv578066910=66, srv673463037=72, srv1945048597=43, srv2135476058=51, srv2009390650=49, srv948355809=98, srv122076318=10, srv377061447=60, srv1387992390=20, srv1498816146=25, srv1421282831=21, srv1476788133=24, srv672390567=71, srv759760060=79, srv10364307=2, srv913165760=93, srv932584258=96, srv1729560149=33, srv2006546142=48, srv912439168=92, srv313556931=55, srv1166122104=8, srv438021544=62, srv1279230135=15, srv1278496520=14, srv1195151220=9, srv781404718=83, srv740963470=77, srv687796086=73, srv1569821429=26, srv1038319955=3, srv1959361613=44, srv1804208178=35, srv20035614=46, srv350525406=59, srv1823872841=38, srv945521501=97, srv1807004428=36, srv322238762=56, srv1421992201=22, srv1602833799=28, srv200384619=47, srv1664905839=30, srv623150267=68, srv299790650=54, srv1104564957=6, srv764814827=81, srv109389693=5, srv825228751=89, srv409644759=61, srv696839131=74, srv887838442=90, srv1017677068=1, srv2071995786=50, srv762469662=80, srv1746422084=34, srv645070831=69, srv1065556504=4, srv184464993=39, srv1351508554=18, srv996204978=99, srv1123674319=7, srv559896043=64, srv1387357953=19, srv1999603125=45, srv286624428=53, srv496552156=63, srv696869375=75, srv797824707=86, srv779599341=82, srv897834965=91, srv788358914=85, srv1320898657=17, srv74807007=78, srv1915870072=41, srv668415034=70, srv1226306891=11, srv582462575=67, srv1898865425=40, srv1008561695=0, srv2137846315=52, srv1823529363=37, srv575847746=65, srv143121918=23, srv928403914=95, srv1254416310=13, srv1586398771=27, srv1287349010=16, srv1934249993=42, srv325131401=57, srv1646347881=29, srv333262429=58, srv714457966=76, srv1725237300=32, srv1246178217=12, srv1723046767=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-30 15:53:17,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-30 15:53:17,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-30 15:53:17,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-30 15:53:17,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-30 15:53:17,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-30 15:53:17,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-30 15:53:17,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-30 15:53:17,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-30 15:53:17,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-30 15:53:17,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-30 15:53:17,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-30 15:53:17,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-30 15:53:17,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-30 15:53:17,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-30 15:53:17,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-30 15:53:17,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-30 15:53:17,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-30 15:53:17,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-30 15:53:17,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-30 15:53:17,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-30 15:53:17,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-30 15:53:17,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-30 15:53:17,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-30 15:53:17,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-30 15:53:17,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-30 15:53:17,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-30 15:53:17,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-30 15:53:17,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-30 15:53:17,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-30 15:53:17,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-30 15:53:17,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-30 15:53:17,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-30 15:53:17,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-30 15:53:17,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-30 15:53:17,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-30 15:53:17,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-30 15:53:17,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-30 15:53:17,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-30 15:53:17,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-30 15:53:17,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-30 15:53:17,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-30 15:53:17,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-30 15:53:17,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-30 15:53:17,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-30 15:53:17,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-30 15:53:17,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-30 15:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-30 15:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-30 15:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-30 15:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-30 15:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-30 15:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-30 15:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-30 15:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-30 15:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-30 15:53:17,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-30 15:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-30 15:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-30 15:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-30 15:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-30 15:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-30 15:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-30 15:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-30 15:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-30 15:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-30 15:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-30 15:53:17,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-30 15:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-30 15:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-30 15:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-30 15:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-30 15:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-30 15:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-30 15:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-30 15:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-30 15:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-30 15:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-30 15:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-30 15:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-30 15:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-30 15:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-30 15:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-30 15:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-30 15:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-30 15:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-30 15:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-30 15:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-30 15:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-30 15:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-30 15:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-30 15:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-30 15:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-30 15:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-30 15:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-30 15:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-30 15:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-30 15:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-30 15:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-30 15:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-30 15:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-30 15:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-30 15:53:17,180 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-30 15:53:17,181 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-30 15:53:17,181 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.6334672746871227E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-30 15:53:17,416 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 335 ms to try 5000 different iterations. Found a solution that moves 15 regions; Going from a computed imbalance of 2.6334672746871227E-5 to a new imbalance of 8.89822248421798E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0016666666666666668); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-30 15:53:17,417 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1320898657:83 , srv1421992201:83 , srv1646347881:83 , srv1746422084:83 , srv668415034:83 , srv714457966:83 , srv764814827:83 , srv787366646:83 , srv807290920:83 , srv1008561695:84 , srv1017677068:84 , srv10364307:84 , srv1038319955:84 , srv1065556504:84 , srv109389693:84 , srv1104564957:84 , srv1166122104:84 , srv1195151220:84 , srv122076318:84 , srv1226306891:84 , srv1246178217:84 , srv1278496520:84 , srv1279230135:84 , srv1287349010:84 , srv1351508554:84 , srv1387992390:84 , srv1421282831:84 , srv1476788133:84 , srv1498816146:84 , srv1569821429:84 , srv1602833799:84 , srv1664905839:84 , srv1725237300:84 , srv1729560149:84 , srv1807004428:84 , srv1823529363:84 , srv1823872841:84 , srv184464993:84 , srv1898865425:84 , srv1915870072:84 , srv1934249993:84 , srv1945048597:84 , srv1959361613:84 , srv1999603125:84 , srv20035614:84 , srv2006546142:84 , srv2009390650:84 , srv2071995786:84 , srv2135476058:84 , srv2137846315:84 , srv286624428:84 , srv299790650:84 , srv313556931:84 , srv322238762:84 , srv325131401:84 , srv333262429:84 , srv377061447:84 , srv409644759:84 , srv438021544:84 , srv496552156:84 , srv559896043:84 , srv575847746:84 , srv578066910:84 , srv582462575:84 , srv623150267:84 , srv645070831:84 , srv672390567:84 , srv673463037:84 , srv687796086:84 , srv696839131:84 , srv696869375:84 , srv740963470:84 , srv74807007:84 , srv759760060:84 , srv762469662:84 , srv779599341:84 , srv781404718:84 , srv788358914:84 , srv797824707:84 , srv825228751:84 , srv887838442:84 , srv897834965:84 , srv912439168:84 , srv913165760:84 , srv926238398:84 , srv928403914:84 , srv932584258:84 , srv945521501:84 , srv948355809:84 , srv996204978:84 , srv1123674319:85 , srv1254416310:85 , srv1387357953:85 , srv143121918:85 , srv1586398771:85 , srv1804208178:85 , srv200384619:85 , srv350525406:85 , srv806055536:85 , srv1723046767:684 } 2023-12-30 15:53:17,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-30 15:53:17,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv787366646=84, srv806055536=87, srv926238398=94, srv807290920=88, srv578066910=66, srv673463037=72, srv1945048597=43, srv2135476058=51, srv2009390650=49, srv948355809=98, srv122076318=10, srv377061447=60, srv1387992390=20, srv1498816146=25, srv1421282831=21, srv1476788133=24, srv672390567=71, srv759760060=79, srv10364307=2, srv913165760=93, srv932584258=96, srv1729560149=33, srv2006546142=48, srv912439168=92, srv313556931=55, srv1166122104=8, srv438021544=62, srv1279230135=15, srv1278496520=14, srv1195151220=9, srv781404718=83, srv740963470=77, srv687796086=73, srv1569821429=26, srv1038319955=3, srv1959361613=44, srv1804208178=35, srv20035614=46, srv350525406=59, srv1823872841=38, srv945521501=97, srv1807004428=36, srv322238762=56, srv1421992201=22, srv1602833799=28, srv200384619=47, srv1664905839=30, srv623150267=68, srv299790650=54, srv1104564957=6, srv764814827=81, srv109389693=5, srv825228751=89, srv409644759=61, srv696839131=74, srv887838442=90, srv1017677068=1, srv2071995786=50, srv762469662=80, srv1746422084=34, srv645070831=69, srv1065556504=4, srv184464993=39, srv1351508554=18, srv996204978=99, srv1123674319=7, srv559896043=64, srv1387357953=19, srv1999603125=45, srv286624428=53, srv496552156=63, srv696869375=75, srv797824707=86, srv779599341=82, srv897834965=91, srv788358914=85, srv1320898657=17, srv74807007=78, srv1915870072=41, srv668415034=70, srv1226306891=11, srv582462575=67, srv1898865425=40, srv1008561695=0, srv2137846315=52, srv1823529363=37, srv575847746=65, srv143121918=23, srv928403914=95, srv1254416310=13, srv1586398771=27, srv1287349010=16, srv1934249993=42, srv325131401=57, srv1646347881=29, srv333262429=58, srv714457966=76, srv1725237300=32, srv1246178217=12, srv1723046767=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-30 15:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-30 15:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-30 15:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-30 15:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-30 15:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-30 15:53:17,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-30 15:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-30 15:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-30 15:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-30 15:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-30 15:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-30 15:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-30 15:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-30 15:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-30 15:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-30 15:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-30 15:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-30 15:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-30 15:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-30 15:53:17,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-30 15:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-30 15:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-30 15:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-30 15:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-30 15:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-30 15:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-30 15:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-30 15:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-30 15:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-30 15:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-30 15:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-30 15:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-30 15:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-30 15:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-30 15:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-30 15:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-30 15:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-30 15:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-30 15:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-30 15:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-30 15:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-30 15:53:17,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-30 15:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-30 15:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-30 15:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-30 15:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-30 15:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-30 15:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-30 15:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-30 15:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-30 15:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-30 15:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-30 15:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-30 15:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-30 15:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-30 15:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-30 15:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-30 15:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-30 15:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-30 15:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-30 15:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-30 15:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-30 15:53:17,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-30 15:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-30 15:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-30 15:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-30 15:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-30 15:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-30 15:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-30 15:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-30 15:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-30 15:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-30 15:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-30 15:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-30 15:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-30 15:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-30 15:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-30 15:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-30 15:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-30 15:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-30 15:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-30 15:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-30 15:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-30 15:53:17,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-30 15:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-30 15:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-30 15:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-30 15:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-30 15:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-30 15:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-30 15:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-30 15:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-30 15:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-30 15:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-30 15:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-30 15:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-30 15:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-30 15:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-30 15:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-30 15:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-30 15:53:17,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-30 15:53:17,543 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=8.793190239318795E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-30 15:53:17,544 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1320898657:83 , srv1421992201:83 , srv1646347881:83 , srv1746422084:83 , srv668415034:83 , srv714457966:83 , srv764814827:83 , srv787366646:83 , srv807290920:83 , srv1008561695:84 , srv1017677068:84 , srv10364307:84 , srv1038319955:84 , srv1065556504:84 , srv109389693:84 , srv1104564957:84 , srv1166122104:84 , srv1195151220:84 , srv122076318:84 , srv1226306891:84 , srv1246178217:84 , srv1278496520:84 , srv1279230135:84 , srv1287349010:84 , srv1351508554:84 , srv1387992390:84 , srv1421282831:84 , srv1476788133:84 , srv1498816146:84 , srv1569821429:84 , srv1602833799:84 , srv1664905839:84 , srv1725237300:84 , srv1729560149:84 , srv1807004428:84 , srv1823529363:84 , srv1823872841:84 , srv184464993:84 , srv1898865425:84 , srv1915870072:84 , srv1934249993:84 , srv1945048597:84 , srv1959361613:84 , srv1999603125:84 , srv20035614:84 , srv2006546142:84 , srv2009390650:84 , srv2071995786:84 , srv2135476058:84 , srv2137846315:84 , srv286624428:84 , srv299790650:84 , srv313556931:84 , srv322238762:84 , srv325131401:84 , srv333262429:84 , srv377061447:84 , srv409644759:84 , srv438021544:84 , srv496552156:84 , srv559896043:84 , srv575847746:84 , srv578066910:84 , srv582462575:84 , srv623150267:84 , srv645070831:84 , srv672390567:84 , srv673463037:84 , srv687796086:84 , srv696839131:84 , srv696869375:84 , srv740963470:84 , srv74807007:84 , srv759760060:84 , srv762469662:84 , srv779599341:84 , srv781404718:84 , srv788358914:84 , srv797824707:84 , srv825228751:84 , srv887838442:84 , srv897834965:84 , srv912439168:84 , srv913165760:84 , srv926238398:84 , srv928403914:84 , srv932584258:84 , srv945521501:84 , srv948355809:84 , srv996204978:84 , srv1123674319:85 , srv1254416310:85 , srv1387357953:85 , srv143121918:85 , srv1586398771:85 , srv1804208178:85 , srv200384619:85 , srv350525406:85 , srv806055536:85 , srv1723046767:684 } 2023-12-30 15:53:17,577 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=285 (was 285), ProcessCount=171 (was 171), AvailableMemoryMB=4494 (was 5106) 2023-12-30 15:53:17,584 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=285, ProcessCount=171, AvailableMemoryMB=4493 2023-12-30 15:53:17,585 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-30 15:53:17,585 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-12-30 15:53:17,596 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1020417884:84 , srv103226082:84 , srv1042016019:84 , srv1072576913:84 , srv1088193975:84 , srv1094494279:84 , srv112572880:84 , srv1156967449:84 , srv1170998379:84 , srv120487323:84 , srv1356400660:84 , srv1359494829:84 , srv1442787:84 , srv1504169605:84 , srv1739913251:84 , srv1911690526:84 , srv1932145236:84 , srv2005925267:84 , srv2018648819:84 , srv247362417:84 , srv316052365:84 , srv397193574:84 , srv451571542:84 , srv477132443:84 , srv520239861:84 , srv558366656:84 , srv603757784:84 , srv736261961:84 , srv766580784:84 , srv1758520519:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-12-30 15:53:17,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-30 15:53:17,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv247362417=20, srv1739913251=14, srv103226082=1, srv1356400660=10, srv120487323=9, srv1072576913=3, srv1042016019=2, srv451571542=23, srv766580784=29, srv1020417884=0, srv1442787=12, srv2018648819=19, srv477132443=24, srv112572880=6, srv1359494829=11, srv736261961=28, srv1758520519=15, srv1504169605=13, srv316052365=21, srv1094494279=5, srv1932145236=17, srv397193574=22, srv1156967449=7, srv1170998379=8, srv558366656=26, srv1911690526=16, srv520239861=25, srv1088193975=4, srv2005925267=18, srv603757784=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-30 15:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-30 15:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-30 15:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-30 15:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-30 15:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-30 15:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-30 15:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-30 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-30 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-30 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-30 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-30 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-30 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-30 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-30 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-30 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-30 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-30 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-30 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-30 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-30 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-30 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-30 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-30 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-30 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-30 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-30 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-30 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-30 15:53:17,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-30 15:53:17,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-30 15:53:17,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-30 15:53:17,624 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1589000414336141, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-30 15:53:17,625 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912693732106272, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1589000414336141, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-12-30 15:55:17,614 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5766686 different iterations. Found a solution that moves 1871 regions; Going from a computed imbalance of 0.9912693732106272 to a new imbalance of 4.367007337919408E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.692962962962963, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-30 15:55:17,622 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1020417884:90 , srv103226082:90 , srv1042016019:90 , srv1072576913:90 , srv1088193975:90 , srv1094494279:90 , srv112572880:90 , srv1156967449:90 , srv1170998379:90 , srv120487323:90 , srv1356400660:90 , srv1359494829:90 , srv1442787:90 , srv1504169605:90 , srv1739913251:90 , srv1758520519:90 , srv1911690526:90 , srv1932145236:90 , srv2005925267:90 , srv2018648819:90 , srv247362417:90 , srv316052365:90 , srv397193574:90 , srv451571542:90 , srv477132443:90 , srv520239861:90 , srv558366656:90 , srv603757784:90 , srv736261961:90 , srv766580784:90 } 2023-12-30 15:55:17,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-30 15:55:17,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv247362417=20, srv1739913251=14, srv103226082=1, srv1356400660=10, srv120487323=9, srv1072576913=3, srv1042016019=2, srv451571542=23, srv766580784=29, srv1020417884=0, srv1442787=12, srv2018648819=19, srv477132443=24, srv112572880=6, srv1359494829=11, srv736261961=28, srv1758520519=15, srv1504169605=13, srv316052365=21, srv1094494279=5, srv1932145236=17, srv397193574=22, srv1156967449=7, srv1170998379=8, srv558366656=26, srv1911690526=16, srv520239861=25, srv1088193975=4, srv2005925267=18, srv603757784=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-30 15:55:17,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-30 15:55:17,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-30 15:55:17,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-30 15:55:17,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-30 15:55:17,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-30 15:55:17,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-30 15:55:17,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-30 15:55:17,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-30 15:55:17,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-30 15:55:17,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-30 15:55:17,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-30 15:55:17,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-30 15:55:17,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-30 15:55:17,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-30 15:55:17,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-30 15:55:17,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-30 15:55:17,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-30 15:55:17,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-30 15:55:17,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-30 15:55:17,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-30 15:55:17,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-30 15:55:17,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-30 15:55:17,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-30 15:55:17,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-30 15:55:17,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-30 15:55:17,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-30 15:55:17,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-30 15:55:17,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-30 15:55:17,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-30 15:55:17,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-30 15:55:17,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-30 15:55:17,644 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-30 15:55:17,644 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1020417884:90 , srv103226082:90 , srv1042016019:90 , srv1072576913:90 , srv1088193975:90 , srv1094494279:90 , srv112572880:90 , srv1156967449:90 , srv1170998379:90 , srv120487323:90 , srv1356400660:90 , srv1359494829:90 , srv1442787:90 , srv1504169605:90 , srv1739913251:90 , srv1758520519:90 , srv1911690526:90 , srv1932145236:90 , srv2005925267:90 , srv2018648819:90 , srv247362417:90 , srv316052365:90 , srv397193574:90 , srv451571542:90 , srv477132443:90 , srv520239861:90 , srv558366656:90 , srv603757784:90 , srv736261961:90 , srv766580784:90 } 2023-12-30 15:55:17,662 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=173 (was 285), ProcessCount=169 (was 171), AvailableMemoryMB=4064 (was 4493)