2023-05-07 18:53:37,707 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-05-07 18:53:38,025 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-05-07 18:53:38,145 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-05-07 18:53:38,207 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-07 18:53:38,214 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-07 18:53:38,229 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=408, ProcessCount=171, AvailableMemoryMB=15516 2023-05-07 18:53:38,231 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-07 18:53:38,232 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-07 18:53:38,363 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1012863132:84 , srv1023136104:84 , srv102376098:84 , srv1044511426:84 , srv1112988997:84 , srv1121682251:84 , srv1122756217:84 , srv1162265037:84 , srv1184705782:84 , srv1188869281:84 , srv1195890148:84 , srv1203026170:84 , srv1204472476:84 , srv1284853043:84 , srv1335636289:84 , srv1364305243:84 , srv137698078:84 , srv141538763:84 , srv1416229250:84 , srv1437860868:84 , srv1481845214:84 , srv149035201:84 , srv1506291872:84 , srv1518877824:84 , srv1534044464:84 , srv1550861309:84 , srv1563879785:84 , srv1570958430:84 , srv1616591973:84 , srv164162494:84 , srv1642305866:84 , srv1648697681:84 , srv1672670792:84 , srv1674665922:84 , srv1674841350:84 , srv1679960378:84 , srv1694321814:84 , srv1714614234:84 , srv1715858512:84 , srv1759238036:84 , srv1759800785:84 , srv1776624057:84 , srv1780667663:84 , srv1795475062:84 , srv1815700565:84 , srv1824551886:84 , srv1841986660:84 , srv1887358024:84 , srv1894994473:84 , srv1942200055:84 , srv1958846452:84 , srv1966658772:84 , srv19798621:84 , srv2022549114:84 , srv2025131049:84 , srv203774904:84 , srv2055164175:84 , srv2058677223:84 , srv2087935386:84 , srv2102688389:84 , srv218557517:84 , srv224203416:84 , srv230645174:84 , srv249916551:84 , srv257820018:84 , srv285492890:84 , srv303461976:84 , srv308342295:84 , srv331397777:84 , srv350698044:84 , srv356561061:84 , srv363083244:84 , srv369577583:84 , srv406939809:84 , srv416542388:84 , srv462578368:84 , srv484134458:84 , srv504639311:84 , srv521312714:84 , srv539542464:84 , srv545434350:84 , srv597846537:84 , srv617654351:84 , srv637857379:84 , srv655674781:84 , srv665624061:84 , srv761666176:84 , srv765905345:84 , srv777255336:84 , srv87082314:84 , srv928899958:84 , srv932390036:84 , srv935438729:84 , srv936428815:84 , srv958984496:84 , srv95926979:84 , srv962997385:84 , srv994279098:84 , srv995770506:84 , srv1398206364:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-05-07 18:53:38,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-07 18:53:38,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1012863132=0, srv1694321814=37, srv257820018=65, srv1284853043=13, srv1122756217=6, srv230645174=63, srv1398206364=17, srv462578368=76, srv1044511426=3, srv1195890148=10, srv2087935386=59, srv1506291872=23, srv1121682251=5, srv1481845214=21, srv1674665922=34, srv521312714=79, srv777255336=89, srv1534044464=25, srv1714614234=38, srv87082314=90, srv994279098=98, srv1674841350=35, srv1112988997=4, srv1780667663=43, srv1364305243=15, srv164162494=30, srv2055164175=57, srv1942200055=50, srv1203026170=11, srv936428815=94, srv1023136104=1, srv1550861309=26, srv1616591973=29, srv1759238036=40, srv1416229250=19, srv962997385=97, srv1518877824=24, srv1648697681=32, srv995770506=99, srv303461976=67, srv637857379=84, srv1759800785=41, srv935438729=93, srv331397777=69, srv95926979=96, srv406939809=74, srv1894994473=49, srv249916551=64, srv350698044=70, srv1642305866=31, srv285492890=66, srv665624061=86, srv765905345=88, srv545434350=81, srv539542464=80, srv1966658772=52, srv1335636289=14, srv1887358024=48, srv1570958430=28, srv1815700565=45, srv1841986660=47, srv1715858512=39, srv1204472476=12, srv2058677223=58, srv149035201=22, srv1679960378=36, srv504639311=78, srv2102688389=60, srv484134458=77, srv141538763=18, srv102376098=2, srv1958846452=51, srv1824551886=46, srv655674781=85, srv1776624057=42, srv2025131049=55, srv224203416=62, srv928899958=91, srv1437860868=20, srv19798621=53, srv761666176=87, srv308342295=68, srv597846537=82, srv137698078=16, srv363083244=72, srv416542388=75, srv1563879785=27, srv932390036=92, srv1795475062=44, srv1672670792=33, srv2022549114=54, srv356561061=71, srv958984496=95, srv203774904=56, srv1188869281=9, srv218557517=61, srv1184705782=8, srv369577583=73, srv1162265037=7, srv617654351=83} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-07 18:53:38,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-07 18:53:38,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-07 18:53:38,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-07 18:53:38,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-07 18:53:38,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-07 18:53:38,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-07 18:53:38,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-07 18:53:38,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-07 18:53:38,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-07 18:53:38,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-07 18:53:38,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-07 18:53:38,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-07 18:53:38,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-07 18:53:38,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-07 18:53:38,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-07 18:53:38,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-07 18:53:38,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-07 18:53:38,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-07 18:53:38,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-07 18:53:38,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-07 18:53:38,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-07 18:53:38,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-07 18:53:38,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-07 18:53:38,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-07 18:53:38,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-07 18:53:38,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-07 18:53:38,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-07 18:53:38,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-07 18:53:38,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-07 18:53:38,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-07 18:53:38,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-07 18:53:38,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-07 18:53:38,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-07 18:53:38,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-07 18:53:38,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-07 18:53:38,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-07 18:53:38,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-07 18:53:38,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-07 18:53:38,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-07 18:53:38,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-07 18:53:38,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-07 18:53:38,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-07 18:53:38,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-07 18:53:38,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-07 18:53:38,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-07 18:53:38,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-07 18:53:38,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-07 18:53:38,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-07 18:53:38,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-07 18:53:38,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-07 18:53:38,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-07 18:53:38,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-07 18:53:38,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-07 18:53:38,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-07 18:53:38,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-07 18:53:38,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-07 18:53:38,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-07 18:53:38,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-07 18:53:38,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-07 18:53:38,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-07 18:53:38,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-07 18:53:38,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-07 18:53:38,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-07 18:53:38,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-07 18:53:38,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-07 18:53:38,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-07 18:53:38,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-07 18:53:38,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-07 18:53:38,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-07 18:53:38,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-07 18:53:38,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-07 18:53:38,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-07 18:53:38,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-07 18:53:38,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-07 18:53:38,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-07 18:53:38,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-07 18:53:38,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-07 18:53:38,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-07 18:53:38,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-07 18:53:38,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-07 18:53:38,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-07 18:53:38,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-07 18:53:38,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-07 18:53:38,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-07 18:53:38,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-07 18:53:38,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-07 18:53:38,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-07 18:53:38,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-07 18:53:38,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-07 18:53:38,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-07 18:53:38,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-07 18:53:38,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-07 18:53:38,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-07 18:53:38,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-07 18:53:38,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-07 18:53:38,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-07 18:53:38,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-07 18:53:38,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-07 18:53:38,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-07 18:53:38,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-07 18:53:38,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-07 18:53:38,668 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-07 18:53:38,668 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-07 18:53:38,668 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-07 18:53:39,011 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 503 ms to try 5000 different iterations. Found a solution that moves 4648 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1877330086580007. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040015584942420565, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05559190729887601, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5164444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040015584942420565, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.183, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.25, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-07 18:53:39,030 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv141538763:82 , srv1966658772:82 , srv1162265037:83 , srv1642305866:83 , srv203774904:83 , srv406939809:83 , srv95926979:83 , srv1112988997:84 , srv164162494:84 , srv1780667663:84 , srv1815700565:84 , srv2102688389:84 , srv462578368:84 , srv761666176:84 , srv1023136104:85 , srv1195890148:85 , srv137698078:85 , srv1437860868:85 , srv1481845214:85 , srv1563879785:85 , srv1674665922:85 , srv1759800785:85 , srv224203416:85 , srv257820018:85 , srv303461976:85 , srv504639311:85 , srv545434350:85 , srv617654351:85 , srv935438729:85 , srv102376098:86 , srv1044511426:86 , srv1122756217:86 , srv1204472476:86 , srv1335636289:86 , srv149035201:86 , srv1506291872:86 , srv1550861309:86 , srv1672670792:86 , srv1679960378:86 , srv1714614234:86 , srv1795475062:86 , srv1887358024:86 , srv1894994473:86 , srv1942200055:86 , srv19798621:86 , srv2022549114:86 , srv2058677223:86 , srv285492890:86 , srv308342295:86 , srv331397777:86 , srv356561061:86 , srv363083244:86 , srv416542388:86 , srv521312714:86 , srv765905345:86 , srv87082314:86 , srv1121682251:87 , srv1184705782:87 , srv1284853043:87 , srv1416229250:87 , srv1570958430:87 , srv1694321814:87 , srv1715858512:87 , srv1776624057:87 , srv1824551886:87 , srv1841986660:87 , srv230645174:87 , srv350698044:87 , srv484134458:87 , srv539542464:87 , srv655674781:87 , srv777255336:87 , srv932390036:87 , srv936428815:87 , srv962997385:87 , srv995770506:87 , srv1012863132:88 , srv1188869281:88 , srv1364305243:88 , srv1616591973:88 , srv1759238036:88 , srv2025131049:88 , srv597846537:88 , srv637857379:88 , srv665624061:88 , srv1203026170:89 , srv1518877824:89 , srv1648697681:89 , srv249916551:89 , srv369577583:89 , srv928899958:89 , srv958984496:89 , srv994279098:89 , srv1958846452:90 , srv2055164175:90 , srv1534044464:91 , srv2087935386:91 , srv218557517:91 , srv1674841350:93 , srv1398206364:446 } 2023-05-07 18:53:39,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-07 18:53:39,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1012863132=0, srv1694321814=37, srv257820018=65, srv1284853043=13, srv1122756217=6, srv230645174=63, srv1398206364=17, srv462578368=76, srv1044511426=3, srv1195890148=10, srv2087935386=59, srv1506291872=23, srv1121682251=5, srv1481845214=21, srv1674665922=34, srv521312714=79, srv777255336=89, srv1534044464=25, srv1714614234=38, srv87082314=90, srv994279098=98, srv1674841350=35, srv1112988997=4, srv1780667663=43, srv1364305243=15, srv164162494=30, srv2055164175=57, srv1942200055=50, srv1203026170=11, srv936428815=94, srv1023136104=1, srv1550861309=26, srv1616591973=29, srv1759238036=40, srv1416229250=19, srv962997385=97, srv1518877824=24, srv1648697681=32, srv995770506=99, srv303461976=67, srv637857379=84, srv1759800785=41, srv935438729=93, srv331397777=69, srv95926979=96, srv406939809=74, srv1894994473=49, srv249916551=64, srv350698044=70, srv1642305866=31, srv285492890=66, srv665624061=86, srv765905345=88, srv545434350=81, srv539542464=80, srv1966658772=52, srv1335636289=14, srv1887358024=48, srv1570958430=28, srv1815700565=45, srv1841986660=47, srv1715858512=39, srv1204472476=12, srv2058677223=58, srv149035201=22, srv1679960378=36, srv504639311=78, srv2102688389=60, srv484134458=77, srv141538763=18, srv102376098=2, srv1958846452=51, srv1824551886=46, srv655674781=85, srv1776624057=42, srv2025131049=55, srv224203416=62, srv928899958=91, srv1437860868=20, srv19798621=53, srv761666176=87, srv308342295=68, srv597846537=82, srv137698078=16, srv363083244=72, srv416542388=75, srv1563879785=27, srv932390036=92, srv1795475062=44, srv1672670792=33, srv2022549114=54, srv356561061=71, srv958984496=95, srv203774904=56, srv1188869281=9, srv218557517=61, srv1184705782=8, srv369577583=73, srv1162265037=7, srv617654351=83} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-07 18:53:39,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-07 18:53:39,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-07 18:53:39,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-07 18:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-07 18:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-07 18:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-07 18:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-07 18:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-07 18:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-07 18:53:39,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-07 18:53:39,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-07 18:53:39,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-07 18:53:39,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-07 18:53:39,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-07 18:53:39,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-07 18:53:39,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-07 18:53:39,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-07 18:53:39,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-07 18:53:39,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-07 18:53:39,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-07 18:53:39,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-07 18:53:39,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-07 18:53:39,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-07 18:53:39,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-07 18:53:39,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-07 18:53:39,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-07 18:53:39,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-07 18:53:39,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-07 18:53:39,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-07 18:53:39,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-07 18:53:39,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-07 18:53:39,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-07 18:53:39,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-07 18:53:39,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-07 18:53:39,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-07 18:53:39,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-07 18:53:39,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-07 18:53:39,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-07 18:53:39,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-07 18:53:39,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-07 18:53:39,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-07 18:53:39,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-07 18:53:39,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-07 18:53:39,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-07 18:53:39,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-07 18:53:39,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-07 18:53:39,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-07 18:53:39,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-07 18:53:39,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-07 18:53:39,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-07 18:53:39,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-07 18:53:39,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-07 18:53:39,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-07 18:53:39,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-07 18:53:39,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-07 18:53:39,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-07 18:53:39,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-07 18:53:39,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-07 18:53:39,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-07 18:53:39,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-07 18:53:39,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-07 18:53:39,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-07 18:53:39,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-07 18:53:39,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-07 18:53:39,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-07 18:53:39,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-07 18:53:39,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-07 18:53:39,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-07 18:53:39,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-07 18:53:39,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-07 18:53:39,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-07 18:53:39,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-07 18:53:39,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-07 18:53:39,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-07 18:53:39,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-07 18:53:39,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-07 18:53:39,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-07 18:53:39,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-07 18:53:39,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-07 18:53:39,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-07 18:53:39,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-07 18:53:39,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-07 18:53:39,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-07 18:53:39,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-07 18:53:39,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-07 18:53:39,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-07 18:53:39,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-07 18:53:39,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-07 18:53:39,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-07 18:53:39,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-07 18:53:39,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-07 18:53:39,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-07 18:53:39,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-07 18:53:39,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-07 18:53:39,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-07 18:53:39,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-07 18:53:39,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-07 18:53:39,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-07 18:53:39,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-07 18:53:39,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-07 18:53:39,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-07 18:53:39,254 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040015584942420565, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05559190729887601, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040015584942420565, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.183, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.25, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-07 18:53:39,255 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-07 18:53:39,255 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18770046266638127, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040015584942420565, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05559190729887601, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040015584942420565, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.183, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.25, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-07 18:53:39,545 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 441 ms to try 5000 different iterations. Found a solution that moves 2670 regions; Going from a computed imbalance of 0.18770046266638127 to a new imbalance of 0.015598691643785394. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020944763726812875); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032080846245810944, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2966666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020944763726812875); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.050333333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-07 18:53:39,558 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1674841350:77 , srv2055164175:81 , srv218557517:81 , srv1203026170:82 , srv2087935386:82 , srv637857379:82 , srv928899958:82 , srv994279098:82 , srv1364305243:83 , srv1534044464:83 , srv1648697681:83 , srv2025131049:83 , srv369577583:83 , srv958984496:83 , srv1012863132:84 , srv1121682251:84 , srv1616591973:84 , srv1715858512:84 , srv1824551886:84 , srv19798621:84 , srv331397777:84 , srv350698044:84 , srv484134458:84 , srv597846537:84 , srv665624061:84 , srv765905345:84 , srv936428815:84 , srv995770506:84 , srv1023136104:85 , srv1122756217:85 , srv1188869281:85 , srv1284853043:85 , srv1335636289:85 , srv1437860868:85 , srv149035201:85 , srv1518877824:85 , srv1550861309:85 , srv1674665922:85 , srv1694321814:85 , srv1714614234:85 , srv1759238036:85 , srv1759800785:85 , srv1776624057:85 , srv1795475062:85 , srv1887358024:85 , srv1942200055:85 , srv1958846452:85 , srv249916551:85 , srv257820018:85 , srv308342295:85 , srv363083244:85 , srv655674781:85 , srv932390036:85 , srv102376098:86 , srv1044511426:86 , srv1112988997:86 , srv1481845214:86 , srv1563879785:86 , srv1570958430:86 , srv1672670792:86 , srv1841986660:86 , srv1894994473:86 , srv224203416:86 , srv230645174:86 , srv356561061:86 , srv416542388:86 , srv617654351:86 , srv87082314:86 , srv935438729:86 , srv1204472476:87 , srv1416229250:87 , srv1506291872:87 , srv2102688389:87 , srv504639311:87 , srv539542464:87 , srv962997385:87 , srv137698078:88 , srv1679960378:88 , srv1780667663:88 , srv2022549114:88 , srv2058677223:88 , srv303461976:88 , srv462578368:88 , srv521312714:88 , srv545434350:88 , srv777255336:88 , srv1162265037:89 , srv1815700565:89 , srv285492890:89 , srv1195890148:90 , srv141538763:90 , srv164162494:90 , srv1642305866:90 , srv406939809:90 , srv761666176:90 , srv1966658772:91 , srv203774904:91 , srv1184705782:92 , srv95926979:92 , srv1398206364:514 } 2023-05-07 18:53:39,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-07 18:53:39,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1012863132=0, srv1694321814=37, srv257820018=65, srv1284853043=13, srv1122756217=6, srv230645174=63, srv1398206364=17, srv462578368=76, srv1044511426=3, srv1195890148=10, srv2087935386=59, srv1506291872=23, srv1121682251=5, srv1481845214=21, srv1674665922=34, srv521312714=79, srv777255336=89, srv1534044464=25, srv1714614234=38, srv87082314=90, srv994279098=98, srv1674841350=35, srv1112988997=4, srv1780667663=43, srv1364305243=15, srv164162494=30, srv2055164175=57, srv1942200055=50, srv1203026170=11, srv936428815=94, srv1023136104=1, srv1550861309=26, srv1616591973=29, srv1759238036=40, srv1416229250=19, srv962997385=97, srv1518877824=24, srv1648697681=32, srv995770506=99, srv303461976=67, srv637857379=84, srv1759800785=41, srv935438729=93, srv331397777=69, srv95926979=96, srv406939809=74, srv1894994473=49, srv249916551=64, srv350698044=70, srv1642305866=31, srv285492890=66, srv665624061=86, srv765905345=88, srv545434350=81, srv539542464=80, srv1966658772=52, srv1335636289=14, srv1887358024=48, srv1570958430=28, srv1815700565=45, srv1841986660=47, srv1715858512=39, srv1204472476=12, srv2058677223=58, srv149035201=22, srv1679960378=36, srv504639311=78, srv2102688389=60, srv484134458=77, srv141538763=18, srv102376098=2, srv1958846452=51, srv1824551886=46, srv655674781=85, srv1776624057=42, srv2025131049=55, srv224203416=62, srv928899958=91, srv1437860868=20, srv19798621=53, srv761666176=87, srv308342295=68, srv597846537=82, srv137698078=16, srv363083244=72, srv416542388=75, srv1563879785=27, srv932390036=92, srv1795475062=44, srv1672670792=33, srv2022549114=54, srv356561061=71, srv958984496=95, srv203774904=56, srv1188869281=9, srv218557517=61, srv1184705782=8, srv369577583=73, srv1162265037=7, srv617654351=83} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-07 18:53:39,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-07 18:53:39,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-07 18:53:39,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-07 18:53:39,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-07 18:53:39,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-07 18:53:39,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-07 18:53:39,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-07 18:53:39,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-07 18:53:39,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-07 18:53:39,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-07 18:53:39,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-07 18:53:39,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-07 18:53:39,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-07 18:53:39,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-07 18:53:39,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-07 18:53:39,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-07 18:53:39,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-07 18:53:39,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-07 18:53:39,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-07 18:53:39,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-07 18:53:39,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-07 18:53:39,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-07 18:53:39,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-07 18:53:39,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-07 18:53:39,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-07 18:53:39,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-07 18:53:39,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-07 18:53:39,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-07 18:53:39,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-07 18:53:39,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-07 18:53:39,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-07 18:53:39,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-07 18:53:39,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-07 18:53:39,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-07 18:53:39,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-07 18:53:39,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-07 18:53:39,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-07 18:53:39,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-07 18:53:39,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-07 18:53:39,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-07 18:53:39,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-07 18:53:39,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-07 18:53:39,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-07 18:53:39,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-07 18:53:39,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-07 18:53:39,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-07 18:53:39,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-07 18:53:39,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-07 18:53:39,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-07 18:53:39,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-07 18:53:39,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-07 18:53:39,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-07 18:53:39,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-07 18:53:39,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-07 18:53:39,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-07 18:53:39,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-07 18:53:39,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-07 18:53:39,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-07 18:53:39,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-07 18:53:39,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-07 18:53:39,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-07 18:53:39,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-07 18:53:39,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-07 18:53:39,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-07 18:53:39,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-07 18:53:39,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-07 18:53:39,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-07 18:53:39,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-07 18:53:39,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-07 18:53:39,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-07 18:53:39,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-07 18:53:39,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-07 18:53:39,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-07 18:53:39,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-07 18:53:39,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-07 18:53:39,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-07 18:53:39,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-07 18:53:39,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-07 18:53:39,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-07 18:53:39,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-07 18:53:39,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-07 18:53:39,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-07 18:53:39,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-07 18:53:39,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-07 18:53:39,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-07 18:53:39,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-07 18:53:39,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-07 18:53:39,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-07 18:53:39,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-07 18:53:39,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-07 18:53:39,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-07 18:53:39,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-07 18:53:39,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-07 18:53:39,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-07 18:53:39,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-07 18:53:39,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-07 18:53:39,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-07 18:53:39,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-07 18:53:39,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-07 18:53:39,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-07 18:53:39,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-07 18:53:39,757 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020944763726812875); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032080846245810944, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020944763726812875); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.050333333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-07 18:53:39,757 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-07 18:53:39,757 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01557999590419334, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020944763726812875); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032080846245810944, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020944763726812875); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.050333333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-07 18:53:40,048 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 430 ms to try 5000 different iterations. Found a solution that moves 1123 regions; Going from a computed imbalance of 0.01557999590419334 to a new imbalance of 7.299030879862834E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004450052578466472); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007309112127323451); MoveCostFunction : (multiplier=7.0, imbalance=0.12477777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004450052578466472); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-07 18:53:40,050 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1184705782:78 , srv1679960378:82 , srv1958846452:82 , srv95926979:82 , srv1195890148:83 , srv203774904:83 , srv249916551:83 , srv285492890:83 , srv521312714:83 , srv777255336:83 , srv1012863132:84 , srv1416229250:84 , srv1518877824:84 , srv1534044464:84 , srv164162494:84 , srv1642305866:84 , srv1759238036:84 , srv1776624057:84 , srv1815700565:84 , srv2022549114:84 , srv2058677223:84 , srv230645174:84 , srv303461976:84 , srv539542464:84 , srv545434350:84 , srv962997385:84 , srv1044511426:85 , srv1188869281:85 , srv1204472476:85 , srv1284853043:85 , srv137698078:85 , srv1481845214:85 , srv1506291872:85 , srv1570958430:85 , srv1616591973:85 , srv1672670792:85 , srv1780667663:85 , srv1841986660:85 , srv1894994473:85 , srv1966658772:85 , srv2087935386:85 , srv356561061:85 , srv406939809:85 , srv462578368:85 , srv504639311:85 , srv761666176:85 , srv935438729:85 , srv102376098:86 , srv141538763:86 , srv1550861309:86 , srv1563879785:86 , srv1648697681:86 , srv1694321814:86 , srv1795475062:86 , srv2025131049:86 , srv2102688389:86 , srv218557517:86 , srv224203416:86 , srv308342295:86 , srv369577583:86 , srv416542388:86 , srv484134458:86 , srv597846537:86 , srv655674781:86 , srv665624061:86 , srv87082314:86 , srv932390036:86 , srv936428815:86 , srv995770506:86 , srv1023136104:87 , srv1112988997:87 , srv1121682251:87 , srv1162265037:87 , srv1335636289:87 , srv1437860868:87 , srv149035201:87 , srv1674665922:87 , srv1674841350:87 , srv1714614234:87 , srv1715858512:87 , srv1759800785:87 , srv1824551886:87 , srv1887358024:87 , srv2055164175:87 , srv257820018:87 , srv350698044:87 , srv363083244:87 , srv928899958:87 , srv958984496:87 , srv994279098:87 , srv1122756217:88 , srv1203026170:88 , srv1364305243:88 , srv1942200055:88 , srv19798621:88 , srv331397777:88 , srv617654351:88 , srv637857379:88 , srv765905345:89 , srv1398206364:537 } 2023-05-07 18:53:40,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-07 18:53:40,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1012863132=0, srv1694321814=37, srv257820018=65, srv1284853043=13, srv1122756217=6, srv230645174=63, srv1398206364=17, srv462578368=76, srv1044511426=3, srv1195890148=10, srv2087935386=59, srv1506291872=23, srv1121682251=5, srv1481845214=21, srv1674665922=34, srv521312714=79, srv777255336=89, srv1534044464=25, srv1714614234=38, srv87082314=90, srv994279098=98, srv1674841350=35, srv1112988997=4, srv1780667663=43, srv1364305243=15, srv164162494=30, srv2055164175=57, srv1942200055=50, srv1203026170=11, srv936428815=94, srv1023136104=1, srv1550861309=26, srv1616591973=29, srv1759238036=40, srv1416229250=19, srv962997385=97, srv1518877824=24, srv1648697681=32, srv995770506=99, srv303461976=67, srv637857379=84, srv1759800785=41, srv935438729=93, srv331397777=69, srv95926979=96, srv406939809=74, srv1894994473=49, srv249916551=64, srv350698044=70, srv1642305866=31, srv285492890=66, srv665624061=86, srv765905345=88, srv545434350=81, srv539542464=80, srv1966658772=52, srv1335636289=14, srv1887358024=48, srv1570958430=28, srv1815700565=45, srv1841986660=47, srv1715858512=39, srv1204472476=12, srv2058677223=58, srv149035201=22, srv1679960378=36, srv504639311=78, srv2102688389=60, srv484134458=77, srv141538763=18, srv102376098=2, srv1958846452=51, srv1824551886=46, srv655674781=85, srv1776624057=42, srv2025131049=55, srv224203416=62, srv928899958=91, srv1437860868=20, srv19798621=53, srv761666176=87, srv308342295=68, srv597846537=82, srv137698078=16, srv363083244=72, srv416542388=75, srv1563879785=27, srv932390036=92, srv1795475062=44, srv1672670792=33, srv2022549114=54, srv356561061=71, srv958984496=95, srv203774904=56, srv1188869281=9, srv218557517=61, srv1184705782=8, srv369577583=73, srv1162265037=7, srv617654351=83} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-07 18:53:40,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-07 18:53:40,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-07 18:53:40,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-07 18:53:40,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-07 18:53:40,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-07 18:53:40,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-07 18:53:40,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-07 18:53:40,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-07 18:53:40,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-07 18:53:40,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-07 18:53:40,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-07 18:53:40,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-07 18:53:40,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-07 18:53:40,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-07 18:53:40,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-07 18:53:40,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-07 18:53:40,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-07 18:53:40,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-07 18:53:40,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-07 18:53:40,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-07 18:53:40,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-07 18:53:40,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-07 18:53:40,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-07 18:53:40,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-07 18:53:40,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-07 18:53:40,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-07 18:53:40,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-07 18:53:40,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-07 18:53:40,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-07 18:53:40,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-07 18:53:40,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-07 18:53:40,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-07 18:53:40,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-07 18:53:40,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-07 18:53:40,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-07 18:53:40,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-07 18:53:40,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-07 18:53:40,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-07 18:53:40,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-07 18:53:40,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-07 18:53:40,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-07 18:53:40,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-07 18:53:40,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-07 18:53:40,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-07 18:53:40,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-07 18:53:40,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-07 18:53:40,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-07 18:53:40,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-07 18:53:40,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-07 18:53:40,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-07 18:53:40,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-07 18:53:40,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-07 18:53:40,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-07 18:53:40,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-07 18:53:40,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-07 18:53:40,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-07 18:53:40,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-07 18:53:40,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-07 18:53:40,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-07 18:53:40,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-07 18:53:40,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-07 18:53:40,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-07 18:53:40,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-07 18:53:40,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-07 18:53:40,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-07 18:53:40,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-07 18:53:40,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-07 18:53:40,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-07 18:53:40,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-07 18:53:40,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-07 18:53:40,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-07 18:53:40,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-07 18:53:40,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-07 18:53:40,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-07 18:53:40,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-07 18:53:40,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-07 18:53:40,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-07 18:53:40,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-07 18:53:40,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-07 18:53:40,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-07 18:53:40,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-07 18:53:40,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-07 18:53:40,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-07 18:53:40,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-07 18:53:40,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-07 18:53:40,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-07 18:53:40,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-07 18:53:40,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-07 18:53:40,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-07 18:53:40,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-07 18:53:40,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-07 18:53:40,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-07 18:53:40,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-07 18:53:40,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-07 18:53:40,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-07 18:53:40,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-07 18:53:40,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-07 18:53:40,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-07 18:53:40,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-07 18:53:40,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-07 18:53:40,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-07 18:53:40,188 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004450052578466472); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007309112127323451); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004450052578466472); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-07 18:53:40,188 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-07 18:53:40,188 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.220396739181645E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004450052578466472); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007309112127323451); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004450052578466472); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-07 18:53:40,691 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 606 ms to try 5000 different iterations. Found a solution that moves 251 regions; Going from a computed imbalance of 7.220396739181645E-4 to a new imbalance of 2.75111278710773E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.02788888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-07 18:53:40,693 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1121682251:83 , srv1122756217:83 , srv1162265037:83 , srv1364305243:83 , srv1942200055:83 , srv406939809:83 , srv617654351:83 , srv761666176:83 , srv765905345:83 , srv777255336:83 , srv958984496:83 , srv102376098:84 , srv1188869281:84 , srv1195890148:84 , srv1203026170:84 , srv1204472476:84 , srv1335636289:84 , srv137698078:84 , srv141538763:84 , srv1416229250:84 , srv149035201:84 , srv1506291872:84 , srv1518877824:84 , srv1534044464:84 , srv1570958430:84 , srv1648697681:84 , srv1694321814:84 , srv1714614234:84 , srv1715858512:84 , srv1815700565:84 , srv1824551886:84 , srv1841986660:84 , srv1887358024:84 , srv1966658772:84 , srv19798621:84 , srv2022549114:84 , srv2055164175:84 , srv2058677223:84 , srv2087935386:84 , srv218557517:84 , srv285492890:84 , srv331397777:84 , srv350698044:84 , srv363083244:84 , srv369577583:84 , srv416542388:84 , srv539542464:84 , srv597846537:84 , srv637857379:84 , srv655674781:84 , srv87082314:84 , srv928899958:84 , srv932390036:84 , srv962997385:84 , srv994279098:84 , srv1023136104:85 , srv1044511426:85 , srv1112988997:85 , srv1184705782:85 , srv1284853043:85 , srv1437860868:85 , srv1550861309:85 , srv1563879785:85 , srv1616591973:85 , srv164162494:85 , srv1642305866:85 , srv1672670792:85 , srv1674665922:85 , srv1674841350:85 , srv1759238036:85 , srv1759800785:85 , srv1780667663:85 , srv1795475062:85 , srv1894994473:85 , srv1958846452:85 , srv203774904:85 , srv2102688389:85 , srv224203416:85 , srv230645174:85 , srv249916551:85 , srv257820018:85 , srv303461976:85 , srv308342295:85 , srv462578368:85 , srv484134458:85 , srv504639311:85 , srv521312714:85 , srv545434350:85 , srv665624061:85 , srv936428815:85 , srv95926979:85 , srv995770506:85 , srv1012863132:86 , srv1481845214:86 , srv1679960378:86 , srv1776624057:86 , srv2025131049:86 , srv356561061:86 , srv935438729:86 , srv1398206364:644 } 2023-05-07 18:53:40,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-07 18:53:40,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1012863132=0, srv1694321814=37, srv257820018=65, srv1284853043=13, srv1122756217=6, srv230645174=63, srv1398206364=17, srv462578368=76, srv1044511426=3, srv1195890148=10, srv2087935386=59, srv1506291872=23, srv1121682251=5, srv1481845214=21, srv1674665922=34, srv521312714=79, srv777255336=89, srv1534044464=25, srv1714614234=38, srv87082314=90, srv994279098=98, srv1674841350=35, srv1112988997=4, srv1780667663=43, srv1364305243=15, srv164162494=30, srv2055164175=57, srv1942200055=50, srv1203026170=11, srv936428815=94, srv1023136104=1, srv1550861309=26, srv1616591973=29, srv1759238036=40, srv1416229250=19, srv962997385=97, srv1518877824=24, srv1648697681=32, srv995770506=99, srv303461976=67, srv637857379=84, srv1759800785=41, srv935438729=93, srv331397777=69, srv95926979=96, srv406939809=74, srv1894994473=49, srv249916551=64, srv350698044=70, srv1642305866=31, srv285492890=66, srv665624061=86, srv765905345=88, srv545434350=81, srv539542464=80, srv1966658772=52, srv1335636289=14, srv1887358024=48, srv1570958430=28, srv1815700565=45, srv1841986660=47, srv1715858512=39, srv1204472476=12, srv2058677223=58, srv149035201=22, srv1679960378=36, srv504639311=78, srv2102688389=60, srv484134458=77, srv141538763=18, srv102376098=2, srv1958846452=51, srv1824551886=46, srv655674781=85, srv1776624057=42, srv2025131049=55, srv224203416=62, srv928899958=91, srv1437860868=20, srv19798621=53, srv761666176=87, srv308342295=68, srv597846537=82, srv137698078=16, srv363083244=72, srv416542388=75, srv1563879785=27, srv932390036=92, srv1795475062=44, srv1672670792=33, srv2022549114=54, srv356561061=71, srv958984496=95, srv203774904=56, srv1188869281=9, srv218557517=61, srv1184705782=8, srv369577583=73, srv1162265037=7, srv617654351=83} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-07 18:53:40,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-07 18:53:40,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-07 18:53:40,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-07 18:53:40,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-07 18:53:40,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-07 18:53:40,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-07 18:53:40,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-07 18:53:40,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-07 18:53:40,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-07 18:53:40,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-07 18:53:40,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-07 18:53:40,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-07 18:53:40,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-07 18:53:40,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-07 18:53:40,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-07 18:53:40,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-07 18:53:40,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-07 18:53:40,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-07 18:53:40,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-07 18:53:40,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-07 18:53:40,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-07 18:53:40,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-07 18:53:40,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-07 18:53:40,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-07 18:53:40,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-07 18:53:40,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-07 18:53:40,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-07 18:53:40,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-07 18:53:40,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-07 18:53:40,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-07 18:53:40,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-07 18:53:40,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-07 18:53:40,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-07 18:53:40,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-07 18:53:40,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-07 18:53:40,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-07 18:53:40,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-07 18:53:40,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-07 18:53:40,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-07 18:53:40,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-07 18:53:40,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-07 18:53:40,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-07 18:53:40,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-07 18:53:40,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-07 18:53:40,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-07 18:53:40,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-07 18:53:40,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-07 18:53:40,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-07 18:53:40,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-07 18:53:40,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-07 18:53:40,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-07 18:53:40,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-07 18:53:40,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-07 18:53:40,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-07 18:53:40,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-07 18:53:40,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-07 18:53:40,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-07 18:53:40,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-07 18:53:40,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-07 18:53:40,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-07 18:53:40,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-07 18:53:40,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-07 18:53:40,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-07 18:53:40,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-07 18:53:40,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-07 18:53:40,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-07 18:53:40,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-07 18:53:40,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-07 18:53:40,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-07 18:53:40,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-07 18:53:40,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-07 18:53:40,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-07 18:53:40,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-07 18:53:40,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-07 18:53:40,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-07 18:53:40,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-07 18:53:40,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-07 18:53:40,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-07 18:53:40,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-07 18:53:40,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-07 18:53:40,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-07 18:53:40,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-07 18:53:40,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-07 18:53:40,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-07 18:53:40,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-07 18:53:40,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-07 18:53:40,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-07 18:53:40,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-07 18:53:40,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-07 18:53:40,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-07 18:53:40,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-07 18:53:40,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-07 18:53:40,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-07 18:53:40,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-07 18:53:40,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-07 18:53:40,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-07 18:53:40,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-07 18:53:40,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-07 18:53:40,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-07 18:53:40,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-07 18:53:40,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-07 18:53:40,851 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-07 18:53:40,851 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-07 18:53:40,851 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.5753588306430954E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-07 18:53:41,187 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 447 ms to try 5000 different iterations. Found a solution that moves 21 regions; Going from a computed imbalance of 2.5753588306430954E-5 to a new imbalance of 7.534802643171416E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0023333333333333335); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-07 18:53:41,188 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1203026170:83 , srv164162494:83 , srv2025131049:83 , srv356561061:83 , srv665624061:83 , srv1012863132:84 , srv1023136104:84 , srv102376098:84 , srv1044511426:84 , srv1112988997:84 , srv1122756217:84 , srv1162265037:84 , srv1184705782:84 , srv1188869281:84 , srv1195890148:84 , srv1204472476:84 , srv1284853043:84 , srv1335636289:84 , srv1364305243:84 , srv137698078:84 , srv141538763:84 , srv1416229250:84 , srv1437860868:84 , srv1481845214:84 , srv149035201:84 , srv1506291872:84 , srv1518877824:84 , srv1534044464:84 , srv1550861309:84 , srv1563879785:84 , srv1570958430:84 , srv1616591973:84 , srv1642305866:84 , srv1648697681:84 , srv1672670792:84 , srv1674665922:84 , srv1674841350:84 , srv1679960378:84 , srv1694321814:84 , srv1714614234:84 , srv1715858512:84 , srv1759238036:84 , srv1759800785:84 , srv1776624057:84 , srv1780667663:84 , srv1795475062:84 , srv1824551886:84 , srv1841986660:84 , srv1887358024:84 , srv1894994473:84 , srv1942200055:84 , srv1958846452:84 , srv1966658772:84 , srv19798621:84 , srv2022549114:84 , srv203774904:84 , srv2055164175:84 , srv2058677223:84 , srv2087935386:84 , srv2102688389:84 , srv218557517:84 , srv224203416:84 , srv230645174:84 , srv249916551:84 , srv257820018:84 , srv285492890:84 , srv303461976:84 , srv308342295:84 , srv331397777:84 , srv350698044:84 , srv363083244:84 , srv369577583:84 , srv416542388:84 , srv462578368:84 , srv484134458:84 , srv504639311:84 , srv521312714:84 , srv539542464:84 , srv545434350:84 , srv597846537:84 , srv617654351:84 , srv637857379:84 , srv655674781:84 , srv761666176:84 , srv765905345:84 , srv87082314:84 , srv928899958:84 , srv932390036:84 , srv935438729:84 , srv936428815:84 , srv958984496:84 , srv95926979:84 , srv962997385:84 , srv994279098:84 , srv995770506:84 , srv1121682251:85 , srv1815700565:85 , srv406939809:85 , srv777255336:85 , srv1398206364:685 } 2023-05-07 18:53:41,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-07 18:53:41,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1012863132=0, srv1694321814=37, srv257820018=65, srv1284853043=13, srv1122756217=6, srv230645174=63, srv1398206364=17, srv462578368=76, srv1044511426=3, srv1195890148=10, srv2087935386=59, srv1506291872=23, srv1121682251=5, srv1481845214=21, srv1674665922=34, srv521312714=79, srv777255336=89, srv1534044464=25, srv1714614234=38, srv87082314=90, srv994279098=98, srv1674841350=35, srv1112988997=4, srv1780667663=43, srv1364305243=15, srv164162494=30, srv2055164175=57, srv1942200055=50, srv1203026170=11, srv936428815=94, srv1023136104=1, srv1550861309=26, srv1616591973=29, srv1759238036=40, srv1416229250=19, srv962997385=97, srv1518877824=24, srv1648697681=32, srv995770506=99, srv303461976=67, srv637857379=84, srv1759800785=41, srv935438729=93, srv331397777=69, srv95926979=96, srv406939809=74, srv1894994473=49, srv249916551=64, srv350698044=70, srv1642305866=31, srv285492890=66, srv665624061=86, srv765905345=88, srv545434350=81, srv539542464=80, srv1966658772=52, srv1335636289=14, srv1887358024=48, srv1570958430=28, srv1815700565=45, srv1841986660=47, srv1715858512=39, srv1204472476=12, srv2058677223=58, srv149035201=22, srv1679960378=36, srv504639311=78, srv2102688389=60, srv484134458=77, srv141538763=18, srv102376098=2, srv1958846452=51, srv1824551886=46, srv655674781=85, srv1776624057=42, srv2025131049=55, srv224203416=62, srv928899958=91, srv1437860868=20, srv19798621=53, srv761666176=87, srv308342295=68, srv597846537=82, srv137698078=16, srv363083244=72, srv416542388=75, srv1563879785=27, srv932390036=92, srv1795475062=44, srv1672670792=33, srv2022549114=54, srv356561061=71, srv958984496=95, srv203774904=56, srv1188869281=9, srv218557517=61, srv1184705782=8, srv369577583=73, srv1162265037=7, srv617654351=83} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-07 18:53:41,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-07 18:53:41,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-07 18:53:41,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-07 18:53:41,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-07 18:53:41,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-07 18:53:41,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-07 18:53:41,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-07 18:53:41,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-07 18:53:41,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-07 18:53:41,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-07 18:53:41,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-07 18:53:41,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-07 18:53:41,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-07 18:53:41,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-07 18:53:41,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-07 18:53:41,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-07 18:53:41,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-07 18:53:41,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-07 18:53:41,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-07 18:53:41,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-07 18:53:41,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-07 18:53:41,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-07 18:53:41,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-07 18:53:41,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-07 18:53:41,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-07 18:53:41,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-07 18:53:41,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-07 18:53:41,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-07 18:53:41,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-07 18:53:41,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-07 18:53:41,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-07 18:53:41,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-07 18:53:41,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-07 18:53:41,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-07 18:53:41,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-07 18:53:41,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-07 18:53:41,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-07 18:53:41,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-07 18:53:41,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-07 18:53:41,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-07 18:53:41,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-07 18:53:41,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-07 18:53:41,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-07 18:53:41,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-07 18:53:41,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-07 18:53:41,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-07 18:53:41,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-07 18:53:41,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-07 18:53:41,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-07 18:53:41,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-07 18:53:41,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-07 18:53:41,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-07 18:53:41,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-07 18:53:41,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-07 18:53:41,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-07 18:53:41,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-07 18:53:41,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-07 18:53:41,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-07 18:53:41,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-07 18:53:41,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-07 18:53:41,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-07 18:53:41,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-07 18:53:41,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-07 18:53:41,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-07 18:53:41,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-07 18:53:41,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-07 18:53:41,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-07 18:53:41,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-07 18:53:41,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-07 18:53:41,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-07 18:53:41,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-07 18:53:41,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-07 18:53:41,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-07 18:53:41,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-07 18:53:41,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-07 18:53:41,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-07 18:53:41,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-07 18:53:41,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-07 18:53:41,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-07 18:53:41,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-07 18:53:41,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-07 18:53:41,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-07 18:53:41,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-07 18:53:41,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-07 18:53:41,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-07 18:53:41,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-07 18:53:41,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-07 18:53:41,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-07 18:53:41,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-07 18:53:41,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-07 18:53:41,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-07 18:53:41,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-07 18:53:41,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-07 18:53:41,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-07 18:53:41,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-07 18:53:41,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-07 18:53:41,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-07 18:53:41,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-07 18:53:41,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-07 18:53:41,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-07 18:53:41,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-07 18:53:41,360 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-07 18:53:41,361 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1203026170:83 , srv164162494:83 , srv2025131049:83 , srv356561061:83 , srv665624061:83 , srv1012863132:84 , srv1023136104:84 , srv102376098:84 , srv1044511426:84 , srv1112988997:84 , srv1122756217:84 , srv1162265037:84 , srv1184705782:84 , srv1188869281:84 , srv1195890148:84 , srv1204472476:84 , srv1284853043:84 , srv1335636289:84 , srv1364305243:84 , srv137698078:84 , srv141538763:84 , srv1416229250:84 , srv1437860868:84 , srv1481845214:84 , srv149035201:84 , srv1506291872:84 , srv1518877824:84 , srv1534044464:84 , srv1550861309:84 , srv1563879785:84 , srv1570958430:84 , srv1616591973:84 , srv1642305866:84 , srv1648697681:84 , srv1672670792:84 , srv1674665922:84 , srv1674841350:84 , srv1679960378:84 , srv1694321814:84 , srv1714614234:84 , srv1715858512:84 , srv1759238036:84 , srv1759800785:84 , srv1776624057:84 , srv1780667663:84 , srv1795475062:84 , srv1824551886:84 , srv1841986660:84 , srv1887358024:84 , srv1894994473:84 , srv1942200055:84 , srv1958846452:84 , srv1966658772:84 , srv19798621:84 , srv2022549114:84 , srv203774904:84 , srv2055164175:84 , srv2058677223:84 , srv2087935386:84 , srv2102688389:84 , srv218557517:84 , srv224203416:84 , srv230645174:84 , srv249916551:84 , srv257820018:84 , srv285492890:84 , srv303461976:84 , srv308342295:84 , srv331397777:84 , srv350698044:84 , srv363083244:84 , srv369577583:84 , srv416542388:84 , srv462578368:84 , srv484134458:84 , srv504639311:84 , srv521312714:84 , srv539542464:84 , srv545434350:84 , srv597846537:84 , srv617654351:84 , srv637857379:84 , srv655674781:84 , srv761666176:84 , srv765905345:84 , srv87082314:84 , srv928899958:84 , srv932390036:84 , srv935438729:84 , srv936428815:84 , srv958984496:84 , srv95926979:84 , srv962997385:84 , srv994279098:84 , srv995770506:84 , srv1121682251:85 , srv1815700565:85 , srv406939809:85 , srv777255336:85 , srv1398206364:685 } 2023-05-07 18:53:41,392 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=415 (was 408) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=14649 (was 15516) 2023-05-07 18:53:41,404 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=415, ProcessCount=171, AvailableMemoryMB=14648 2023-05-07 18:53:41,405 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-07 18:53:41,405 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-05-07 18:53:41,417 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1032117448:84 , srv1171852491:84 , srv1316648556:84 , srv1460161813:84 , srv1490626159:84 , srv150150960:84 , srv155816093:84 , srv1606305918:84 , srv1666794949:84 , srv168661804:84 , srv1687266368:84 , srv1800000354:84 , srv1960898996:84 , srv199412940:84 , srv216618365:84 , srv218021914:84 , srv23831561:84 , srv248062534:84 , srv267485639:84 , srv317348264:84 , srv350647045:84 , srv477463233:84 , srv583883419:84 , srv631129531:84 , srv653303087:84 , srv701700177:84 , srv920990787:84 , srv930650554:84 , srv993435649:84 , srv1606942693:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-05-07 18:53:41,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-07 18:53:41,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv993435649=29, srv1316648556=2, srv1171852491=1, srv350647045=21, srv701700177=26, srv930650554=28, srv216618365=15, srv631129531=24, srv1666794949=9, srv653303087=25, srv920990787=27, srv23831561=17, srv267485639=19, srv150150960=5, srv1960898996=13, srv218021914=16, srv1687266368=11, srv583883419=23, srv248062534=18, srv1606305918=7, srv1490626159=4, srv1460161813=3, srv168661804=10, srv1800000354=12, srv1032117448=0, srv317348264=20, srv477463233=22, srv1606942693=8, srv155816093=6, srv199412940=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-07 18:53:41,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-07 18:53:41,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-07 18:53:41,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-07 18:53:41,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-07 18:53:41,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-07 18:53:41,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-07 18:53:41,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-07 18:53:41,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-07 18:53:41,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-07 18:53:41,450 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1780786180611003, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-07 18:53:41,450 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912754163174987, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1780786180611003, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-05-07 18:55:41,435 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6131964 different iterations. Found a solution that moves 1861 regions; Going from a computed imbalance of 0.9912754163174987 to a new imbalance of 4.3436668390529226E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6892592592592592, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-07 18:55:41,444 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1032117448:90 , srv1171852491:90 , srv1316648556:90 , srv1460161813:90 , srv1490626159:90 , srv150150960:90 , srv155816093:90 , srv1606305918:90 , srv1606942693:90 , srv1666794949:90 , srv168661804:90 , srv1687266368:90 , srv1800000354:90 , srv1960898996:90 , srv199412940:90 , srv216618365:90 , srv218021914:90 , srv23831561:90 , srv248062534:90 , srv267485639:90 , srv317348264:90 , srv350647045:90 , srv477463233:90 , srv583883419:90 , srv631129531:90 , srv653303087:90 , srv701700177:90 , srv920990787:90 , srv930650554:90 , srv993435649:90 } 2023-05-07 18:55:41,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-07 18:55:41,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv993435649=29, srv1316648556=2, srv1171852491=1, srv350647045=21, srv701700177=26, srv930650554=28, srv216618365=15, srv631129531=24, srv1666794949=9, srv653303087=25, srv920990787=27, srv23831561=17, srv267485639=19, srv150150960=5, srv1960898996=13, srv218021914=16, srv1687266368=11, srv583883419=23, srv248062534=18, srv1606305918=7, srv1490626159=4, srv1460161813=3, srv168661804=10, srv1800000354=12, srv1032117448=0, srv317348264=20, srv477463233=22, srv1606942693=8, srv155816093=6, srv199412940=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-07 18:55:41,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-07 18:55:41,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-07 18:55:41,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-07 18:55:41,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-07 18:55:41,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-07 18:55:41,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-07 18:55:41,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-07 18:55:41,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-07 18:55:41,462 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-07 18:55:41,463 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1032117448:90 , srv1171852491:90 , srv1316648556:90 , srv1460161813:90 , srv1490626159:90 , srv150150960:90 , srv155816093:90 , srv1606305918:90 , srv1606942693:90 , srv1666794949:90 , srv168661804:90 , srv1687266368:90 , srv1800000354:90 , srv1960898996:90 , srv199412940:90 , srv216618365:90 , srv218021914:90 , srv23831561:90 , srv248062534:90 , srv267485639:90 , srv317348264:90 , srv350647045:90 , srv477463233:90 , srv583883419:90 , srv631129531:90 , srv653303087:90 , srv701700177:90 , srv920990787:90 , srv930650554:90 , srv993435649:90 } 2023-05-07 18:55:41,476 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=204 (was 415), ProcessCount=169 (was 171), AvailableMemoryMB=14600 (was 14648)