2023-08-27 17:53:09,479 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-08-27 17:53:09,801 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-08-27 17:53:09,915 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-08-27 17:53:09,987 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-27 17:53:09,996 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-27 17:53:10,010 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=264, ProcessCount=173, AvailableMemoryMB=13000 2023-08-27 17:53:10,012 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-27 17:53:10,012 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-08-27 17:53:10,174 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1008656841:84 , srv10377669:84 , srv1085657390:84 , srv1103734912:84 , srv1126350316:84 , srv1185630094:84 , srv1203141194:84 , srv1203321164:84 , srv1208575968:84 , srv1215722000:84 , srv1222541602:84 , srv1237702178:84 , srv1243634173:84 , srv1246828797:84 , srv126191236:84 , srv1280289693:84 , srv1283121939:84 , srv1284613967:84 , srv1294272927:84 , srv1317536185:84 , srv1335873825:84 , srv1382320593:84 , srv1389350075:84 , srv1392339790:84 , srv1407041325:84 , srv1422749138:84 , srv1472471535:84 , srv1480929076:84 , srv1496155360:84 , srv1506320006:84 , srv1526554751:84 , srv1548161901:84 , srv1561781031:84 , srv1570762931:84 , srv1575202059:84 , srv1589573717:84 , srv160679231:84 , srv1622564421:84 , srv1628714458:84 , srv1672748732:84 , srv1717740637:84 , srv1787011239:84 , srv1799177818:84 , srv1813240409:84 , srv1825792128:84 , srv1841389530:84 , srv1844384155:84 , srv1846005893:84 , srv1906826334:84 , srv1929613314:84 , srv1934996262:84 , srv1957715244:84 , srv1967189535:84 , srv1974520476:84 , srv1981529033:84 , srv1981532633:84 , srv1997791427:84 , srv2005311934:84 , srv2035176693:84 , srv2050649402:84 , srv2070991244:84 , srv2092774173:84 , srv2131505411:84 , srv2144147475:84 , srv216731457:84 , srv229997062:84 , srv245804653:84 , srv266687632:84 , srv304831975:84 , srv314161199:84 , srv320735196:84 , srv322551167:84 , srv347715586:84 , srv34905327:84 , srv352412863:84 , srv381340387:84 , srv391750671:84 , srv418475098:84 , srv441227524:84 , srv445128099:84 , srv461354537:84 , srv473981052:84 , srv490598480:84 , srv492008310:84 , srv557470520:84 , srv58376370:84 , srv666800381:84 , srv706092539:84 , srv709613112:84 , srv73404881:84 , srv780760168:84 , srv782063156:84 , srv790640672:84 , srv861038887:84 , srv875703793:84 , srv940564709:84 , srv942805243:84 , srv967202114:84 , srv984509800:84 , srv1165526047:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-08-27 17:53:10,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-27 17:53:10,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1846005893=48, srv1526554751=31, srv706092539=88, srv58376370=86, srv229997062=66, srv34905327=74, srv381340387=76, srv1841389530=46, srv1496155360=29, srv1203321164=8, srv1981529033=55, srv1283121939=17, srv1799177818=43, srv216731457=65, srv1622564421=38, srv1825792128=45, srv347715586=73, srv1280289693=16, srv126191236=15, srv304831975=69, srv1246828797=14, srv1103734912=3, srv709613112=89, srv391750671=77, srv352412863=75, srv160679231=37, srv314161199=70, srv942805243=97, srv1165526047=5, srv875703793=95, srv1957715244=52, srv1934996262=51, srv1008656841=0, srv266687632=68, srv322551167=72, srv666800381=87, srv1906826334=49, srv2092774173=62, srv1335873825=21, srv1974520476=54, srv320735196=71, srv984509800=99, srv1787011239=42, srv1185630094=6, srv1317536185=20, srv1422749138=26, srv780760168=91, srv10377669=1, srv1126350316=4, srv1589573717=36, srv1389350075=23, srv782063156=92, srv73404881=90, srv2131505411=63, srv473981052=82, srv1506320006=30, srv2035176693=59, srv1203141194=7, srv790640672=93, srv1981532633=56, srv2070991244=61, srv1243634173=13, srv1208575968=9, srv1085657390=2, srv2050649402=60, srv461354537=81, srv1570762931=34, srv940564709=96, srv418475098=78, srv1222541602=11, srv1382320593=22, srv1844384155=47, srv1237702178=12, srv1480929076=28, srv1628714458=39, srv967202114=98, srv1561781031=33, srv861038887=94, srv1717740637=41, srv1548161901=32, srv1215722000=10, srv445128099=80, srv1392339790=24, srv441227524=79, srv1997791427=57, srv557470520=85, srv1294272927=19, srv2005311934=58, srv245804653=67, srv1407041325=25, srv1813240409=44, srv1575202059=35, srv490598480=83, srv1284613967=18, srv1967189535=53, srv1472471535=27, srv492008310=84, srv1672748732=40, srv2144147475=64, srv1929613314=50} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-27 17:53:10,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-27 17:53:10,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-27 17:53:10,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-27 17:53:10,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-27 17:53:10,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-27 17:53:10,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-27 17:53:10,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-27 17:53:10,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-27 17:53:10,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-27 17:53:10,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-27 17:53:10,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-27 17:53:10,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-27 17:53:10,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-27 17:53:10,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-27 17:53:10,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-27 17:53:10,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-27 17:53:10,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-27 17:53:10,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-27 17:53:10,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-27 17:53:10,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-27 17:53:10,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-27 17:53:10,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-27 17:53:10,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-27 17:53:10,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-27 17:53:10,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-27 17:53:10,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-27 17:53:10,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-27 17:53:10,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-27 17:53:10,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-27 17:53:10,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-27 17:53:10,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-27 17:53:10,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-27 17:53:10,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-27 17:53:10,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-27 17:53:10,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-27 17:53:10,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-27 17:53:10,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-27 17:53:10,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-27 17:53:10,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-27 17:53:10,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-27 17:53:10,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-27 17:53:10,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-27 17:53:10,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-27 17:53:10,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-27 17:53:10,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-27 17:53:10,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-27 17:53:10,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-27 17:53:10,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-27 17:53:10,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-27 17:53:10,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-27 17:53:10,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-27 17:53:10,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-27 17:53:10,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-27 17:53:10,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-27 17:53:10,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-27 17:53:10,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-27 17:53:10,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-27 17:53:10,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-27 17:53:10,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-27 17:53:10,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-27 17:53:10,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-27 17:53:10,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-27 17:53:10,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-27 17:53:10,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-27 17:53:10,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-27 17:53:10,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-27 17:53:10,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-27 17:53:10,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-27 17:53:10,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-27 17:53:10,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-27 17:53:10,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-27 17:53:10,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-27 17:53:10,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-27 17:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-27 17:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-27 17:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-27 17:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-27 17:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-27 17:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-27 17:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-27 17:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-27 17:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-27 17:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-27 17:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-27 17:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-27 17:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-27 17:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-27 17:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-27 17:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-27 17:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-27 17:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-27 17:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-27 17:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-27 17:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-27 17:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-27 17:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-27 17:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-27 17:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-27 17:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-27 17:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-27 17:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-27 17:53:10,506 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-27 17:53:10,506 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-27 17:53:10,507 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-27 17:53:10,911 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 612 ms to try 5000 different iterations. Found a solution that moves 4701 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17677791299141477. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041068470621778354, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05733121912944715, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5223333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041068470621778354, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17175, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24066666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-27 17:53:10,934 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1215722000:81 , srv557470520:81 , srv1589573717:82 , srv1628714458:82 , srv1085657390:83 , srv1185630094:83 , srv1407041325:83 , srv1622564421:83 , srv1981529033:83 , srv2131505411:83 , srv320735196:83 , srv10377669:84 , srv1126350316:84 , srv1203321164:84 , srv1246828797:84 , srv1284613967:84 , srv1317536185:84 , srv1422749138:84 , srv1570762931:84 , srv1841389530:84 , srv1974520476:84 , srv2050649402:84 , srv490598480:84 , srv58376370:84 , srv780760168:84 , srv1103734912:85 , srv1294272927:85 , srv1382320593:85 , srv1480929076:85 , srv1548161901:85 , srv1799177818:85 , srv1934996262:85 , srv1957715244:85 , srv861038887:85 , srv967202114:85 , srv984509800:85 , srv1203141194:86 , srv1392339790:86 , srv1526554751:86 , srv1717740637:86 , srv1813240409:86 , srv1929613314:86 , srv1967189535:86 , srv2035176693:86 , srv2092774173:86 , srv216731457:86 , srv229997062:86 , srv314161199:86 , srv441227524:86 , srv445128099:86 , srv666800381:86 , srv706092539:86 , srv709613112:86 , srv73404881:86 , srv782063156:86 , srv790640672:86 , srv942805243:86 , srv1008656841:87 , srv1208575968:87 , srv1472471535:87 , srv1561781031:87 , srv160679231:87 , srv1672748732:87 , srv1906826334:87 , srv2070991244:87 , srv266687632:87 , srv322551167:87 , srv347715586:87 , srv34905327:87 , srv381340387:87 , srv391750671:87 , srv473981052:87 , srv492008310:87 , srv875703793:87 , srv940564709:87 , srv1237702178:88 , srv126191236:88 , srv1335873825:88 , srv1787011239:88 , srv1981532633:88 , srv2144147475:88 , srv304831975:88 , srv1222541602:89 , srv1283121939:89 , srv1575202059:89 , srv1844384155:89 , srv245804653:89 , srv418475098:89 , srv461354537:89 , srv1389350075:90 , srv1506320006:90 , srv1825792128:90 , srv2005311934:90 , srv352412863:90 , srv1280289693:91 , srv1997791427:91 , srv1846005893:92 , srv1243634173:95 , srv1496155360:97 , srv1165526047:455 } 2023-08-27 17:53:10,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-27 17:53:10,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1846005893=48, srv1526554751=31, srv706092539=88, srv58376370=86, srv229997062=66, srv34905327=74, srv381340387=76, srv1841389530=46, srv1496155360=29, srv1203321164=8, srv1981529033=55, srv1283121939=17, srv1799177818=43, srv216731457=65, srv1622564421=38, srv1825792128=45, srv347715586=73, srv1280289693=16, srv126191236=15, srv304831975=69, srv1246828797=14, srv1103734912=3, srv709613112=89, srv391750671=77, srv352412863=75, srv160679231=37, srv314161199=70, srv942805243=97, srv1165526047=5, srv875703793=95, srv1957715244=52, srv1934996262=51, srv1008656841=0, srv266687632=68, srv322551167=72, srv666800381=87, srv1906826334=49, srv2092774173=62, srv1335873825=21, srv1974520476=54, srv320735196=71, srv984509800=99, srv1787011239=42, srv1185630094=6, srv1317536185=20, srv1422749138=26, srv780760168=91, srv10377669=1, srv1126350316=4, srv1589573717=36, srv1389350075=23, srv782063156=92, srv73404881=90, srv2131505411=63, srv473981052=82, srv1506320006=30, srv2035176693=59, srv1203141194=7, srv790640672=93, srv1981532633=56, srv2070991244=61, srv1243634173=13, srv1208575968=9, srv1085657390=2, srv2050649402=60, srv461354537=81, srv1570762931=34, srv940564709=96, srv418475098=78, srv1222541602=11, srv1382320593=22, srv1844384155=47, srv1237702178=12, srv1480929076=28, srv1628714458=39, srv967202114=98, srv1561781031=33, srv861038887=94, srv1717740637=41, srv1548161901=32, srv1215722000=10, srv445128099=80, srv1392339790=24, srv441227524=79, srv1997791427=57, srv557470520=85, srv1294272927=19, srv2005311934=58, srv245804653=67, srv1407041325=25, srv1813240409=44, srv1575202059=35, srv490598480=83, srv1284613967=18, srv1967189535=53, srv1472471535=27, srv492008310=84, srv1672748732=40, srv2144147475=64, srv1929613314=50} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-27 17:53:10,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-27 17:53:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-27 17:53:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-27 17:53:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-27 17:53:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-27 17:53:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-27 17:53:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-27 17:53:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-27 17:53:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-27 17:53:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-27 17:53:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-27 17:53:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-27 17:53:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-27 17:53:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-27 17:53:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-27 17:53:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-27 17:53:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-27 17:53:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-27 17:53:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-27 17:53:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-27 17:53:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-27 17:53:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-27 17:53:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-27 17:53:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-27 17:53:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-27 17:53:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-27 17:53:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-27 17:53:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-27 17:53:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-27 17:53:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-27 17:53:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-27 17:53:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-27 17:53:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-27 17:53:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-27 17:53:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-27 17:53:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-27 17:53:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-27 17:53:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-27 17:53:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-27 17:53:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-27 17:53:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-27 17:53:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-27 17:53:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-27 17:53:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-27 17:53:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-27 17:53:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-27 17:53:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-27 17:53:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-27 17:53:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-27 17:53:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-27 17:53:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-27 17:53:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-27 17:53:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-27 17:53:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-27 17:53:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-27 17:53:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-27 17:53:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-27 17:53:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-27 17:53:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-27 17:53:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-27 17:53:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-27 17:53:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-27 17:53:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-27 17:53:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-27 17:53:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-27 17:53:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-27 17:53:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-27 17:53:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-27 17:53:10,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-27 17:53:10,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-27 17:53:10,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-27 17:53:10,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-27 17:53:10,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-27 17:53:10,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-27 17:53:10,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-27 17:53:10,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-27 17:53:10,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-27 17:53:10,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-27 17:53:10,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-27 17:53:10,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-27 17:53:10,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-27 17:53:10,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-27 17:53:10,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-27 17:53:10,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-27 17:53:10,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-27 17:53:10,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-27 17:53:10,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-27 17:53:10,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-27 17:53:10,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-27 17:53:10,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-27 17:53:10,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-27 17:53:10,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-27 17:53:10,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-27 17:53:10,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-27 17:53:10,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-27 17:53:10,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-27 17:53:10,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-27 17:53:10,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-27 17:53:10,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-27 17:53:10,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-27 17:53:10,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-27 17:53:11,096 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041068470621778354, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05733121912944715, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041068470621778354, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17175, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24066666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-27 17:53:11,096 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-27 17:53:11,096 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17674499588586337, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041068470621778354, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05733121912944715, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041068470621778354, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17175, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24066666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-27 17:53:11,550 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 558 ms to try 5000 different iterations. Found a solution that moves 2611 regions; Going from a computed imbalance of 0.17674499588586337 to a new imbalance of 0.016793533297140936. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02262375727671853); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03414828691807003, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2901111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02262375727671853); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0135); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04841666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-27 17:53:11,562 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1496155360:75 , srv1243634173:76 , srv1280289693:78 , srv1997791427:79 , srv1846005893:80 , srv2005311934:80 , srv1787011239:81 , srv1506320006:82 , srv1825792128:82 , srv1844384155:82 , srv245804653:82 , srv347715586:82 , srv461354537:82 , srv1283121939:83 , srv1389350075:83 , srv1981532633:83 , srv2144147475:83 , srv304831975:83 , srv1222541602:84 , srv126191236:84 , srv1335873825:84 , srv1526554751:84 , srv266687632:84 , srv352412863:84 , srv666800381:84 , srv709613112:84 , srv942805243:84 , srv1103734912:85 , srv1126350316:85 , srv1237702178:85 , srv1480929076:85 , srv1717740637:85 , srv1906826334:85 , srv1929613314:85 , srv314161199:85 , srv322551167:85 , srv34905327:85 , srv381340387:85 , srv391750671:85 , srv445128099:85 , srv473981052:85 , srv706092539:85 , srv940564709:85 , srv1203141194:86 , srv1382320593:86 , srv1472471535:86 , srv1561781031:86 , srv1570762931:86 , srv1575202059:86 , srv160679231:86 , srv1672748732:86 , srv1813240409:86 , srv1967189535:86 , srv2070991244:86 , srv2092774173:86 , srv320735196:86 , srv418475098:86 , srv441227524:86 , srv492008310:86 , srv782063156:86 , srv875703793:86 , srv967202114:86 , srv1008656841:87 , srv1185630094:87 , srv1208575968:87 , srv1284613967:87 , srv1392339790:87 , srv1407041325:87 , srv1974520476:87 , srv2035176693:87 , srv2050649402:87 , srv490598480:87 , srv73404881:87 , srv790640672:87 , srv861038887:87 , srv1422749138:88 , srv1799177818:88 , srv1841389530:88 , srv1934996262:88 , srv1981529033:88 , srv216731457:88 , srv229997062:88 , srv780760168:88 , srv984509800:88 , srv1215722000:89 , srv1246828797:89 , srv1294272927:89 , srv557470520:89 , srv10377669:90 , srv1085657390:90 , srv1203321164:90 , srv1548161901:90 , srv1628714458:90 , srv1957715244:90 , srv2131505411:90 , srv58376370:90 , srv1317536185:91 , srv1589573717:91 , srv1622564421:93 , srv1165526047:520 } 2023-08-27 17:53:11,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-27 17:53:11,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1846005893=48, srv1526554751=31, srv706092539=88, srv58376370=86, srv229997062=66, srv34905327=74, srv381340387=76, srv1841389530=46, srv1496155360=29, srv1203321164=8, srv1981529033=55, srv1283121939=17, srv1799177818=43, srv216731457=65, srv1622564421=38, srv1825792128=45, srv347715586=73, srv1280289693=16, srv126191236=15, srv304831975=69, srv1246828797=14, srv1103734912=3, srv709613112=89, srv391750671=77, srv352412863=75, srv160679231=37, srv314161199=70, srv942805243=97, srv1165526047=5, srv875703793=95, srv1957715244=52, srv1934996262=51, srv1008656841=0, srv266687632=68, srv322551167=72, srv666800381=87, srv1906826334=49, srv2092774173=62, srv1335873825=21, srv1974520476=54, srv320735196=71, srv984509800=99, srv1787011239=42, srv1185630094=6, srv1317536185=20, srv1422749138=26, srv780760168=91, srv10377669=1, srv1126350316=4, srv1589573717=36, srv1389350075=23, srv782063156=92, srv73404881=90, srv2131505411=63, srv473981052=82, srv1506320006=30, srv2035176693=59, srv1203141194=7, srv790640672=93, srv1981532633=56, srv2070991244=61, srv1243634173=13, srv1208575968=9, srv1085657390=2, srv2050649402=60, srv461354537=81, srv1570762931=34, srv940564709=96, srv418475098=78, srv1222541602=11, srv1382320593=22, srv1844384155=47, srv1237702178=12, srv1480929076=28, srv1628714458=39, srv967202114=98, srv1561781031=33, srv861038887=94, srv1717740637=41, srv1548161901=32, srv1215722000=10, srv445128099=80, srv1392339790=24, srv441227524=79, srv1997791427=57, srv557470520=85, srv1294272927=19, srv2005311934=58, srv245804653=67, srv1407041325=25, srv1813240409=44, srv1575202059=35, srv490598480=83, srv1284613967=18, srv1967189535=53, srv1472471535=27, srv492008310=84, srv1672748732=40, srv2144147475=64, srv1929613314=50} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-27 17:53:11,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-27 17:53:11,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-27 17:53:11,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-27 17:53:11,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-27 17:53:11,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-27 17:53:11,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-27 17:53:11,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-27 17:53:11,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-27 17:53:11,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-27 17:53:11,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-27 17:53:11,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-27 17:53:11,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-27 17:53:11,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-27 17:53:11,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-27 17:53:11,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-27 17:53:11,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-27 17:53:11,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-27 17:53:11,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-27 17:53:11,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-27 17:53:11,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-27 17:53:11,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-27 17:53:11,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-27 17:53:11,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-27 17:53:11,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-27 17:53:11,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-27 17:53:11,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-27 17:53:11,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-27 17:53:11,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-27 17:53:11,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-27 17:53:11,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-27 17:53:11,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-27 17:53:11,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-27 17:53:11,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-27 17:53:11,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-27 17:53:11,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-27 17:53:11,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-27 17:53:11,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-27 17:53:11,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-27 17:53:11,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-27 17:53:11,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-27 17:53:11,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-27 17:53:11,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-27 17:53:11,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-27 17:53:11,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-27 17:53:11,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-27 17:53:11,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-27 17:53:11,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-27 17:53:11,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-27 17:53:11,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-27 17:53:11,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-27 17:53:11,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-27 17:53:11,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-27 17:53:11,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-27 17:53:11,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-27 17:53:11,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-27 17:53:11,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-27 17:53:11,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-27 17:53:11,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-27 17:53:11,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-27 17:53:11,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-27 17:53:11,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-27 17:53:11,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-27 17:53:11,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-27 17:53:11,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-27 17:53:11,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-27 17:53:11,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-27 17:53:11,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-27 17:53:11,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-27 17:53:11,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-27 17:53:11,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-27 17:53:11,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-27 17:53:11,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-27 17:53:11,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-27 17:53:11,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-27 17:53:11,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-27 17:53:11,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-27 17:53:11,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-27 17:53:11,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-27 17:53:11,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-27 17:53:11,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-27 17:53:11,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-27 17:53:11,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-27 17:53:11,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-27 17:53:11,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-27 17:53:11,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-27 17:53:11,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-27 17:53:11,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-27 17:53:11,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-27 17:53:11,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-27 17:53:11,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-27 17:53:11,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-27 17:53:11,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-27 17:53:11,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-27 17:53:11,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-27 17:53:11,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-27 17:53:11,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-27 17:53:11,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-27 17:53:11,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-27 17:53:11,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-27 17:53:11,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-27 17:53:11,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-27 17:53:11,722 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02262375727671853); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03414828691807003, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02262375727671853); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0135); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04841666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-27 17:53:11,723 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-27 17:53:11,723 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01677525068437882, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02262375727671853); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03414828691807003, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02262375727671853); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0135); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04841666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-27 17:53:12,114 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 504 ms to try 5000 different iterations. Found a solution that moves 1035 regions; Going from a computed imbalance of 0.01677525068437882 to a new imbalance of 6.316013302851152E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006009511519817671); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008939251268621247); MoveCostFunction : (multiplier=7.0, imbalance=0.115, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006009511519817671); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0045); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-27 17:53:12,116 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1317536185:82 , srv1622564421:82 , srv216731457:82 , srv1008656841:83 , srv1208575968:83 , srv1294272927:83 , srv1548161901:83 , srv1575202059:83 , srv1957715244:83 , srv418475098:83 , srv58376370:83 , srv10377669:84 , srv1085657390:84 , srv1203141194:84 , srv1203321164:84 , srv1237702178:84 , srv1246828797:84 , srv1389350075:84 , srv1561781031:84 , srv1589573717:84 , srv160679231:84 , srv1672748732:84 , srv1799177818:84 , srv1841389530:84 , srv2070991244:84 , srv229997062:84 , srv352412863:84 , srv492008310:84 , srv73404881:84 , srv875703793:84 , srv984509800:84 , srv1222541602:85 , srv1243634173:85 , srv1283121939:85 , srv1392339790:85 , srv1422749138:85 , srv1506320006:85 , srv1813240409:85 , srv1825792128:85 , srv1906826334:85 , srv1934996262:85 , srv1967189535:85 , srv2035176693:85 , srv304831975:85 , srv34905327:85 , srv381340387:85 , srv473981052:85 , srv790640672:85 , srv861038887:85 , srv940564709:85 , srv126191236:86 , srv1335873825:86 , srv1382320593:86 , srv1496155360:86 , srv1844384155:86 , srv1846005893:86 , srv1929613314:86 , srv1974520476:86 , srv1981532633:86 , srv2050649402:86 , srv2092774173:86 , srv2131505411:86 , srv266687632:86 , srv391750671:86 , srv441227524:86 , srv445128099:86 , srv461354537:86 , srv490598480:86 , srv706092539:86 , srv782063156:86 , srv942805243:86 , srv1103734912:87 , srv1185630094:87 , srv1284613967:87 , srv1472471535:87 , srv1526554751:87 , srv1628714458:87 , srv1717740637:87 , srv1981529033:87 , srv2144147475:87 , srv314161199:87 , srv322551167:87 , srv557470520:87 , srv709613112:87 , srv780760168:87 , srv967202114:87 , srv1215722000:88 , srv1407041325:88 , srv1480929076:88 , srv1787011239:88 , srv1997791427:88 , srv2005311934:88 , srv320735196:88 , srv347715586:88 , srv666800381:88 , srv1126350316:89 , srv1280289693:89 , srv1570762931:89 , srv245804653:89 , srv1165526047:536 } 2023-08-27 17:53:12,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-27 17:53:12,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1846005893=48, srv1526554751=31, srv706092539=88, srv58376370=86, srv229997062=66, srv34905327=74, srv381340387=76, srv1841389530=46, srv1496155360=29, srv1203321164=8, srv1981529033=55, srv1283121939=17, srv1799177818=43, srv216731457=65, srv1622564421=38, srv1825792128=45, srv347715586=73, srv1280289693=16, srv126191236=15, srv304831975=69, srv1246828797=14, srv1103734912=3, srv709613112=89, srv391750671=77, srv352412863=75, srv160679231=37, srv314161199=70, srv942805243=97, srv1165526047=5, srv875703793=95, srv1957715244=52, srv1934996262=51, srv1008656841=0, srv266687632=68, srv322551167=72, srv666800381=87, srv1906826334=49, srv2092774173=62, srv1335873825=21, srv1974520476=54, srv320735196=71, srv984509800=99, srv1787011239=42, srv1185630094=6, srv1317536185=20, srv1422749138=26, srv780760168=91, srv10377669=1, srv1126350316=4, srv1589573717=36, srv1389350075=23, srv782063156=92, srv73404881=90, srv2131505411=63, srv473981052=82, srv1506320006=30, srv2035176693=59, srv1203141194=7, srv790640672=93, srv1981532633=56, srv2070991244=61, srv1243634173=13, srv1208575968=9, srv1085657390=2, srv2050649402=60, srv461354537=81, srv1570762931=34, srv940564709=96, srv418475098=78, srv1222541602=11, srv1382320593=22, srv1844384155=47, srv1237702178=12, srv1480929076=28, srv1628714458=39, srv967202114=98, srv1561781031=33, srv861038887=94, srv1717740637=41, srv1548161901=32, srv1215722000=10, srv445128099=80, srv1392339790=24, srv441227524=79, srv1997791427=57, srv557470520=85, srv1294272927=19, srv2005311934=58, srv245804653=67, srv1407041325=25, srv1813240409=44, srv1575202059=35, srv490598480=83, srv1284613967=18, srv1967189535=53, srv1472471535=27, srv492008310=84, srv1672748732=40, srv2144147475=64, srv1929613314=50} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-27 17:53:12,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-27 17:53:12,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-27 17:53:12,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-27 17:53:12,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-27 17:53:12,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-27 17:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-27 17:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-27 17:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-27 17:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-27 17:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-27 17:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-27 17:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-27 17:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-27 17:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-27 17:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-27 17:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-27 17:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-27 17:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-27 17:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-27 17:53:12,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-27 17:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-27 17:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-27 17:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-27 17:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-27 17:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-27 17:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-27 17:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-27 17:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-27 17:53:12,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-27 17:53:12,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-27 17:53:12,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-27 17:53:12,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-27 17:53:12,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-27 17:53:12,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-27 17:53:12,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-27 17:53:12,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-27 17:53:12,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-27 17:53:12,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-27 17:53:12,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-27 17:53:12,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-27 17:53:12,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-27 17:53:12,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-27 17:53:12,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-27 17:53:12,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-27 17:53:12,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-27 17:53:12,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-27 17:53:12,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-27 17:53:12,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-27 17:53:12,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-27 17:53:12,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-27 17:53:12,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-27 17:53:12,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-27 17:53:12,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-27 17:53:12,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-27 17:53:12,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-27 17:53:12,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-27 17:53:12,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-27 17:53:12,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-27 17:53:12,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-27 17:53:12,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-27 17:53:12,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-27 17:53:12,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-27 17:53:12,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-27 17:53:12,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-27 17:53:12,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-27 17:53:12,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-27 17:53:12,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-27 17:53:12,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-27 17:53:12,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-27 17:53:12,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-27 17:53:12,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-27 17:53:12,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-27 17:53:12,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-27 17:53:12,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-27 17:53:12,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-27 17:53:12,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-27 17:53:12,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-27 17:53:12,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-27 17:53:12,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-27 17:53:12,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-27 17:53:12,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-27 17:53:12,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-27 17:53:12,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-27 17:53:12,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-27 17:53:12,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-27 17:53:12,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-27 17:53:12,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-27 17:53:12,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-27 17:53:12,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-27 17:53:12,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-27 17:53:12,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-27 17:53:12,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-27 17:53:12,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-27 17:53:12,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-27 17:53:12,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-27 17:53:12,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-27 17:53:12,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-27 17:53:12,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-27 17:53:12,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-27 17:53:12,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-27 17:53:12,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-27 17:53:12,313 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006009511519817671); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008939251268621247); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006009511519817671); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0045); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-27 17:53:12,313 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-27 17:53:12,313 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.243541053870715E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006009511519817671); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008939251268621247); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006009511519817671); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0045); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-27 17:53:12,697 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 526 ms to try 5000 different iterations. Found a solution that moves 269 regions; Going from a computed imbalance of 6.243541053870715E-4 to a new imbalance of 4.4863446621686744E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023210354127426377); MoveCostFunction : (multiplier=7.0, imbalance=0.02988888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-27 17:53:12,699 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1472471535:82 , srv245804653:82 , srv1215722000:83 , srv1407041325:83 , srv1570762931:83 , srv1628714458:83 , srv2131505411:83 , srv391750671:83 , srv780760168:83 , srv10377669:84 , srv1203321164:84 , srv1222541602:84 , srv126191236:84 , srv1280289693:84 , srv1284613967:84 , srv1335873825:84 , srv1382320593:84 , srv1392339790:84 , srv1496155360:84 , srv1548161901:84 , srv1575202059:84 , srv160679231:84 , srv1622564421:84 , srv1717740637:84 , srv1846005893:84 , srv1934996262:84 , srv1957715244:84 , srv1967189535:84 , srv1981529033:84 , srv1997791427:84 , srv2035176693:84 , srv2092774173:84 , srv2144147475:84 , srv229997062:84 , srv266687632:84 , srv314161199:84 , srv322551167:84 , srv347715586:84 , srv352412863:84 , srv418475098:84 , srv441227524:84 , srv666800381:84 , srv709613112:84 , srv782063156:84 , srv790640672:84 , srv861038887:84 , srv967202114:84 , srv1008656841:85 , srv1085657390:85 , srv1103734912:85 , srv1126350316:85 , srv1185630094:85 , srv1208575968:85 , srv1237702178:85 , srv1246828797:85 , srv1283121939:85 , srv1294272927:85 , srv1317536185:85 , srv1389350075:85 , srv1422749138:85 , srv1480929076:85 , srv1506320006:85 , srv1526554751:85 , srv1561781031:85 , srv1589573717:85 , srv1672748732:85 , srv1787011239:85 , srv1813240409:85 , srv1825792128:85 , srv1906826334:85 , srv1929613314:85 , srv1974520476:85 , srv2005311934:85 , srv2050649402:85 , srv2070991244:85 , srv320735196:85 , srv34905327:85 , srv381340387:85 , srv445128099:85 , srv461354537:85 , srv473981052:85 , srv490598480:85 , srv492008310:85 , srv557470520:85 , srv58376370:85 , srv73404881:85 , srv942805243:85 , srv984509800:85 , srv1243634173:86 , srv1799177818:86 , srv1841389530:86 , srv1844384155:86 , srv1981532633:86 , srv216731457:86 , srv304831975:86 , srv706092539:86 , srv875703793:86 , srv940564709:86 , srv1203141194:87 , srv1165526047:631 } 2023-08-27 17:53:12,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-27 17:53:12,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1846005893=48, srv1526554751=31, srv706092539=88, srv58376370=86, srv229997062=66, srv34905327=74, srv381340387=76, srv1841389530=46, srv1496155360=29, srv1203321164=8, srv1981529033=55, srv1283121939=17, srv1799177818=43, srv216731457=65, srv1622564421=38, srv1825792128=45, srv347715586=73, srv1280289693=16, srv126191236=15, srv304831975=69, srv1246828797=14, srv1103734912=3, srv709613112=89, srv391750671=77, srv352412863=75, srv160679231=37, srv314161199=70, srv942805243=97, srv1165526047=5, srv875703793=95, srv1957715244=52, srv1934996262=51, srv1008656841=0, srv266687632=68, srv322551167=72, srv666800381=87, srv1906826334=49, srv2092774173=62, srv1335873825=21, srv1974520476=54, srv320735196=71, srv984509800=99, srv1787011239=42, srv1185630094=6, srv1317536185=20, srv1422749138=26, srv780760168=91, srv10377669=1, srv1126350316=4, srv1589573717=36, srv1389350075=23, srv782063156=92, srv73404881=90, srv2131505411=63, srv473981052=82, srv1506320006=30, srv2035176693=59, srv1203141194=7, srv790640672=93, srv1981532633=56, srv2070991244=61, srv1243634173=13, srv1208575968=9, srv1085657390=2, srv2050649402=60, srv461354537=81, srv1570762931=34, srv940564709=96, srv418475098=78, srv1222541602=11, srv1382320593=22, srv1844384155=47, srv1237702178=12, srv1480929076=28, srv1628714458=39, srv967202114=98, srv1561781031=33, srv861038887=94, srv1717740637=41, srv1548161901=32, srv1215722000=10, srv445128099=80, srv1392339790=24, srv441227524=79, srv1997791427=57, srv557470520=85, srv1294272927=19, srv2005311934=58, srv245804653=67, srv1407041325=25, srv1813240409=44, srv1575202059=35, srv490598480=83, srv1284613967=18, srv1967189535=53, srv1472471535=27, srv492008310=84, srv1672748732=40, srv2144147475=64, srv1929613314=50} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-27 17:53:12,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-27 17:53:12,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-27 17:53:12,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-27 17:53:12,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-27 17:53:12,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-27 17:53:12,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-27 17:53:12,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-27 17:53:12,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-27 17:53:12,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-27 17:53:12,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-27 17:53:12,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-27 17:53:12,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-27 17:53:12,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-27 17:53:12,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-27 17:53:12,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-27 17:53:12,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-27 17:53:12,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-27 17:53:12,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-27 17:53:12,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-27 17:53:12,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-27 17:53:12,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-27 17:53:12,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-27 17:53:12,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-27 17:53:12,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-27 17:53:12,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-27 17:53:12,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-27 17:53:12,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-27 17:53:12,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-27 17:53:12,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-27 17:53:12,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-27 17:53:12,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-27 17:53:12,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-27 17:53:12,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-27 17:53:12,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-27 17:53:12,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-27 17:53:12,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-27 17:53:12,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-27 17:53:12,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-27 17:53:12,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-27 17:53:12,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-27 17:53:12,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-27 17:53:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-27 17:53:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-27 17:53:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-27 17:53:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-27 17:53:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-27 17:53:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-27 17:53:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-27 17:53:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-27 17:53:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-27 17:53:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-27 17:53:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-27 17:53:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-27 17:53:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-27 17:53:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-27 17:53:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-27 17:53:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-27 17:53:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-27 17:53:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-27 17:53:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-27 17:53:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-27 17:53:12,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-27 17:53:12,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-27 17:53:12,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-27 17:53:12,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-27 17:53:12,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-27 17:53:12,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-27 17:53:12,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-27 17:53:12,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-27 17:53:12,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-27 17:53:12,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-27 17:53:12,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-27 17:53:12,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-27 17:53:12,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-27 17:53:12,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-27 17:53:12,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-27 17:53:12,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-27 17:53:12,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-27 17:53:12,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-27 17:53:12,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-27 17:53:12,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-27 17:53:12,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-27 17:53:12,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-27 17:53:12,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-27 17:53:12,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-27 17:53:12,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-27 17:53:12,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-27 17:53:12,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-27 17:53:12,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-27 17:53:12,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-27 17:53:12,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-27 17:53:12,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-27 17:53:12,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-27 17:53:12,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-27 17:53:12,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-27 17:53:12,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-27 17:53:12,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-27 17:53:12,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-27 17:53:12,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-27 17:53:12,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-27 17:53:12,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-27 17:53:12,849 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023210354127426377); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-27 17:53:12,849 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-08-27 17:53:12,849 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.2979868363161375E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023210354127426377); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-08-27 17:53:13,186 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 449 ms to try 5000 different iterations. Found a solution that moves 37 regions; Going from a computed imbalance of 4.2979868363161375E-5 to a new imbalance of 8.238769654458575E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.004111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-27 17:53:13,187 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1126350316:83 , srv1203141194:83 , srv1480929076:83 , srv1799177818:83 , srv1844384155:83 , srv1981532633:83 , srv2005311934:83 , srv322551167:83 , srv706092539:83 , srv875703793:83 , srv940564709:83 , srv1008656841:84 , srv10377669:84 , srv1085657390:84 , srv1103734912:84 , srv1185630094:84 , srv1203321164:84 , srv1208575968:84 , srv1222541602:84 , srv1237702178:84 , srv1243634173:84 , srv1246828797:84 , srv126191236:84 , srv1280289693:84 , srv1283121939:84 , srv1284613967:84 , srv1294272927:84 , srv1317536185:84 , srv1335873825:84 , srv1389350075:84 , srv1392339790:84 , srv1422749138:84 , srv1472471535:84 , srv1496155360:84 , srv1506320006:84 , srv1526554751:84 , srv1548161901:84 , srv1561781031:84 , srv1570762931:84 , srv1575202059:84 , srv1589573717:84 , srv1622564421:84 , srv1628714458:84 , srv1672748732:84 , srv1717740637:84 , srv1787011239:84 , srv1813240409:84 , srv1825792128:84 , srv1841389530:84 , srv1846005893:84 , srv1906826334:84 , srv1929613314:84 , srv1934996262:84 , srv1957715244:84 , srv1974520476:84 , srv1981529033:84 , srv1997791427:84 , srv2035176693:84 , srv2050649402:84 , srv2070991244:84 , srv2092774173:84 , srv2131505411:84 , srv2144147475:84 , srv216731457:84 , srv229997062:84 , srv245804653:84 , srv304831975:84 , srv314161199:84 , srv320735196:84 , srv34905327:84 , srv352412863:84 , srv381340387:84 , srv418475098:84 , srv441227524:84 , srv445128099:84 , srv461354537:84 , srv473981052:84 , srv490598480:84 , srv492008310:84 , srv557470520:84 , srv58376370:84 , srv666800381:84 , srv73404881:84 , srv780760168:84 , srv782063156:84 , srv790640672:84 , srv967202114:84 , srv984509800:84 , srv1215722000:85 , srv1382320593:85 , srv1407041325:85 , srv160679231:85 , srv1967189535:85 , srv266687632:85 , srv347715586:85 , srv391750671:85 , srv709613112:85 , srv861038887:85 , srv942805243:85 , srv1165526047:684 } 2023-08-27 17:53:13,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-27 17:53:13,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1846005893=48, srv1526554751=31, srv706092539=88, srv58376370=86, srv229997062=66, srv34905327=74, srv381340387=76, srv1841389530=46, srv1496155360=29, srv1203321164=8, srv1981529033=55, srv1283121939=17, srv1799177818=43, srv216731457=65, srv1622564421=38, srv1825792128=45, srv347715586=73, srv1280289693=16, srv126191236=15, srv304831975=69, srv1246828797=14, srv1103734912=3, srv709613112=89, srv391750671=77, srv352412863=75, srv160679231=37, srv314161199=70, srv942805243=97, srv1165526047=5, srv875703793=95, srv1957715244=52, srv1934996262=51, srv1008656841=0, srv266687632=68, srv322551167=72, srv666800381=87, srv1906826334=49, srv2092774173=62, srv1335873825=21, srv1974520476=54, srv320735196=71, srv984509800=99, srv1787011239=42, srv1185630094=6, srv1317536185=20, srv1422749138=26, srv780760168=91, srv10377669=1, srv1126350316=4, srv1589573717=36, srv1389350075=23, srv782063156=92, srv73404881=90, srv2131505411=63, srv473981052=82, srv1506320006=30, srv2035176693=59, srv1203141194=7, srv790640672=93, srv1981532633=56, srv2070991244=61, srv1243634173=13, srv1208575968=9, srv1085657390=2, srv2050649402=60, srv461354537=81, srv1570762931=34, srv940564709=96, srv418475098=78, srv1222541602=11, srv1382320593=22, srv1844384155=47, srv1237702178=12, srv1480929076=28, srv1628714458=39, srv967202114=98, srv1561781031=33, srv861038887=94, srv1717740637=41, srv1548161901=32, srv1215722000=10, srv445128099=80, srv1392339790=24, srv441227524=79, srv1997791427=57, srv557470520=85, srv1294272927=19, srv2005311934=58, srv245804653=67, srv1407041325=25, srv1813240409=44, srv1575202059=35, srv490598480=83, srv1284613967=18, srv1967189535=53, srv1472471535=27, srv492008310=84, srv1672748732=40, srv2144147475=64, srv1929613314=50} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-27 17:53:13,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-27 17:53:13,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-27 17:53:13,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-27 17:53:13,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-27 17:53:13,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-27 17:53:13,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-27 17:53:13,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-27 17:53:13,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-27 17:53:13,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-27 17:53:13,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-27 17:53:13,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-27 17:53:13,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-27 17:53:13,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-27 17:53:13,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-27 17:53:13,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-27 17:53:13,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-27 17:53:13,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-27 17:53:13,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-27 17:53:13,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-27 17:53:13,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-27 17:53:13,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-27 17:53:13,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-27 17:53:13,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-27 17:53:13,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-27 17:53:13,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-27 17:53:13,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-27 17:53:13,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-27 17:53:13,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-27 17:53:13,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-27 17:53:13,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-27 17:53:13,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-08-27 17:53:13,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-08-27 17:53:13,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-08-27 17:53:13,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-08-27 17:53:13,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-08-27 17:53:13,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-08-27 17:53:13,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-08-27 17:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-08-27 17:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-08-27 17:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-08-27 17:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-08-27 17:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-08-27 17:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-08-27 17:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-08-27 17:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-08-27 17:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-08-27 17:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-08-27 17:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-08-27 17:53:13,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-08-27 17:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-08-27 17:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-08-27 17:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-08-27 17:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-08-27 17:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-08-27 17:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-08-27 17:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-08-27 17:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-08-27 17:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-08-27 17:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-08-27 17:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-08-27 17:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-08-27 17:53:13,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-08-27 17:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-08-27 17:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-08-27 17:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-08-27 17:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-08-27 17:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-08-27 17:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-08-27 17:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-08-27 17:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-08-27 17:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-08-27 17:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-08-27 17:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-08-27 17:53:13,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-08-27 17:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-08-27 17:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-08-27 17:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-08-27 17:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-08-27 17:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-08-27 17:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-08-27 17:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-08-27 17:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-08-27 17:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-08-27 17:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-08-27 17:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-08-27 17:53:13,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-08-27 17:53:13,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-08-27 17:53:13,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-08-27 17:53:13,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-08-27 17:53:13,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-08-27 17:53:13,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-08-27 17:53:13,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-08-27 17:53:13,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-08-27 17:53:13,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-08-27 17:53:13,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-08-27 17:53:13,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-08-27 17:53:13,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-08-27 17:53:13,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-08-27 17:53:13,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-08-27 17:53:13,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-08-27 17:53:13,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-08-27 17:53:13,363 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.979690117040589E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-27 17:53:13,363 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1126350316:83 , srv1203141194:83 , srv1480929076:83 , srv1799177818:83 , srv1844384155:83 , srv1981532633:83 , srv2005311934:83 , srv322551167:83 , srv706092539:83 , srv875703793:83 , srv940564709:83 , srv1008656841:84 , srv10377669:84 , srv1085657390:84 , srv1103734912:84 , srv1185630094:84 , srv1203321164:84 , srv1208575968:84 , srv1222541602:84 , srv1237702178:84 , srv1243634173:84 , srv1246828797:84 , srv126191236:84 , srv1280289693:84 , srv1283121939:84 , srv1284613967:84 , srv1294272927:84 , srv1317536185:84 , srv1335873825:84 , srv1389350075:84 , srv1392339790:84 , srv1422749138:84 , srv1472471535:84 , srv1496155360:84 , srv1506320006:84 , srv1526554751:84 , srv1548161901:84 , srv1561781031:84 , srv1570762931:84 , srv1575202059:84 , srv1589573717:84 , srv1622564421:84 , srv1628714458:84 , srv1672748732:84 , srv1717740637:84 , srv1787011239:84 , srv1813240409:84 , srv1825792128:84 , srv1841389530:84 , srv1846005893:84 , srv1906826334:84 , srv1929613314:84 , srv1934996262:84 , srv1957715244:84 , srv1974520476:84 , srv1981529033:84 , srv1997791427:84 , srv2035176693:84 , srv2050649402:84 , srv2070991244:84 , srv2092774173:84 , srv2131505411:84 , srv2144147475:84 , srv216731457:84 , srv229997062:84 , srv245804653:84 , srv304831975:84 , srv314161199:84 , srv320735196:84 , srv34905327:84 , srv352412863:84 , srv381340387:84 , srv418475098:84 , srv441227524:84 , srv445128099:84 , srv461354537:84 , srv473981052:84 , srv490598480:84 , srv492008310:84 , srv557470520:84 , srv58376370:84 , srv666800381:84 , srv73404881:84 , srv780760168:84 , srv782063156:84 , srv790640672:84 , srv967202114:84 , srv984509800:84 , srv1215722000:85 , srv1382320593:85 , srv1407041325:85 , srv160679231:85 , srv1967189535:85 , srv266687632:85 , srv347715586:85 , srv391750671:85 , srv709613112:85 , srv861038887:85 , srv942805243:85 , srv1165526047:684 } 2023-08-27 17:53:13,410 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=307 (was 264) - SystemLoadAverage LEAK? -, ProcessCount=173 (was 173), AvailableMemoryMB=12124 (was 13000) 2023-08-27 17:53:13,429 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=265, MaxFileDescriptor=60000, SystemLoadAverage=307, ProcessCount=173, AvailableMemoryMB=12123 2023-08-27 17:53:13,430 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-08-27 17:53:13,430 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-08-27 17:53:13,447 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1123967198:84 , srv1279383775:84 , srv1387419990:84 , srv1399687236:84 , srv1447330101:84 , srv1455453247:84 , srv1477876333:84 , srv1599493680:84 , srv1660562562:84 , srv1690582852:84 , srv1803257893:84 , srv182754505:84 , srv1961178670:84 , srv2145015388:84 , srv2145098365:84 , srv246729727:84 , srv256706066:84 , srv270839013:84 , srv291206840:84 , srv366849913:84 , srv374266233:84 , srv470844707:84 , srv532999723:84 , srv602204199:84 , srv680613818:84 , srv709385470:84 , srv757055802:84 , srv832026045:84 , srv860095710:84 , srv771996773:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-08-27 17:53:13,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-27 17:53:13,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1455453247=5, srv532999723=22, srv1279383775=1, srv1447330101=4, srv1477876333=6, srv1803257893=10, srv270839013=17, srv470844707=21, srv291206840=18, srv602204199=23, srv771996773=27, srv680613818=24, srv1123967198=0, srv182754505=11, srv1387419990=2, srv757055802=26, srv2145098365=14, srv1599493680=7, srv366849913=19, srv860095710=29, srv709385470=25, srv1961178670=12, srv256706066=16, srv832026045=28, srv1690582852=9, srv374266233=20, srv1399687236=3, srv2145015388=13, srv246729727=15, srv1660562562=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-27 17:53:13,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-27 17:53:13,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-27 17:53:13,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-27 17:53:13,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-27 17:53:13,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-27 17:53:13,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-27 17:53:13,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-27 17:53:13,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-27 17:53:13,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-27 17:53:13,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-27 17:53:13,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-27 17:53:13,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-27 17:53:13,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-27 17:53:13,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-27 17:53:13,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-27 17:53:13,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-27 17:53:13,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-27 17:53:13,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-27 17:53:13,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-27 17:53:13,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-27 17:53:13,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-27 17:53:13,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-27 17:53:13,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-27 17:53:13,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-27 17:53:13,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-27 17:53:13,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-27 17:53:13,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-27 17:53:13,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-27 17:53:13,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-27 17:53:13,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-27 17:53:13,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-27 17:53:13,500 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1065001897020013, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-27 17:53:13,500 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.991252862188448, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1065001897020013, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-08-27 17:55:13,472 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5590468 different iterations. Found a solution that moves 1853 regions; Going from a computed imbalance of 0.991252862188448 to a new imbalance of 4.324994439959734E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6862962962962963, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-27 17:55:13,478 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1123967198:90 , srv1279383775:90 , srv1387419990:90 , srv1399687236:90 , srv1447330101:90 , srv1455453247:90 , srv1477876333:90 , srv1599493680:90 , srv1660562562:90 , srv1690582852:90 , srv1803257893:90 , srv182754505:90 , srv1961178670:90 , srv2145015388:90 , srv2145098365:90 , srv246729727:90 , srv256706066:90 , srv270839013:90 , srv291206840:90 , srv366849913:90 , srv374266233:90 , srv470844707:90 , srv532999723:90 , srv602204199:90 , srv680613818:90 , srv709385470:90 , srv757055802:90 , srv771996773:90 , srv832026045:90 , srv860095710:90 } 2023-08-27 17:55:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-08-27 17:55:13,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1455453247=5, srv532999723=22, srv1279383775=1, srv1447330101=4, srv1477876333=6, srv1803257893=10, srv270839013=17, srv470844707=21, srv291206840=18, srv602204199=23, srv771996773=27, srv680613818=24, srv1123967198=0, srv182754505=11, srv1387419990=2, srv757055802=26, srv2145098365=14, srv1599493680=7, srv366849913=19, srv860095710=29, srv709385470=25, srv1961178670=12, srv256706066=16, srv832026045=28, srv1690582852=9, srv374266233=20, srv1399687236=3, srv2145015388=13, srv246729727=15, srv1660562562=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-08-27 17:55:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-08-27 17:55:13,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-08-27 17:55:13,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-08-27 17:55:13,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-08-27 17:55:13,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-08-27 17:55:13,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-08-27 17:55:13,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-08-27 17:55:13,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-08-27 17:55:13,495 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-08-27 17:55:13,495 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1123967198:90 , srv1279383775:90 , srv1387419990:90 , srv1399687236:90 , srv1447330101:90 , srv1455453247:90 , srv1477876333:90 , srv1599493680:90 , srv1660562562:90 , srv1690582852:90 , srv1803257893:90 , srv182754505:90 , srv1961178670:90 , srv2145015388:90 , srv2145098365:90 , srv246729727:90 , srv256706066:90 , srv270839013:90 , srv291206840:90 , srv366849913:90 , srv374266233:90 , srv470844707:90 , srv532999723:90 , srv602204199:90 , srv680613818:90 , srv709385470:90 , srv757055802:90 , srv771996773:90 , srv832026045:90 , srv860095710:90 } 2023-08-27 17:55:13,507 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 265), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=222 (was 307), ProcessCount=169 (was 173), AvailableMemoryMB=12062 (was 12123)