2023-03-31 23:00:58,503 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-03-31 23:00:58,905 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-31 23:00:59,027 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-31 23:00:59,078 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-31 23:00:59,085 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-31 23:00:59,098 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=280, ProcessCount=164, AvailableMemoryMB=9398 2023-03-31 23:00:59,100 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-31 23:00:59,101 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-31 23:00:59,240 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1014483938:84 , srv1021506254:84 , srv108070015:84 , srv1084073672:84 , srv109052039:84 , srv1108796757:84 , srv1162789374:84 , srv120302473:84 , srv1224185724:84 , srv1235397417:84 , srv1239336532:84 , srv1244401102:84 , srv1255005705:84 , srv1277114570:84 , srv1281656784:84 , srv1313921885:84 , srv1363437273:84 , srv137858070:84 , srv1407047253:84 , srv1411383544:84 , srv148267407:84 , srv1486002140:84 , srv1489382946:84 , srv1494769812:84 , srv1495154121:84 , srv1516801510:84 , srv1522712963:84 , srv1527638091:84 , srv1544015565:84 , srv1566761471:84 , srv1578972575:84 , srv1585280640:84 , srv1592112089:84 , srv1633378885:84 , srv1641643155:84 , srv1642721394:84 , srv1662821621:84 , srv167487081:84 , srv1722597674:84 , srv1726376418:84 , srv1782469508:84 , srv1784843618:84 , srv1788087587:84 , srv1807075525:84 , srv1832844274:84 , srv1869218624:84 , srv1920695380:84 , srv1931854191:84 , srv194681324:84 , srv1963205277:84 , srv1969824445:84 , srv1998571452:84 , srv2022721097:84 , srv2026133758:84 , srv2027271655:84 , srv2032526871:84 , srv2036072612:84 , srv2053665836:84 , srv2055432822:84 , srv2059711284:84 , srv2063117797:84 , srv2079013959:84 , srv2106071889:84 , srv2118781429:84 , srv236186232:84 , srv262358735:84 , srv310979086:84 , srv316988670:84 , srv334263920:84 , srv34731578:84 , srv374079553:84 , srv414481684:84 , srv451227853:84 , srv497634472:84 , srv498968558:84 , srv521375047:84 , srv545380744:84 , srv556269136:84 , srv565166447:84 , srv599062518:84 , srv60615455:84 , srv615663105:84 , srv623828435:84 , srv65281114:84 , srv659772160:84 , srv66035524:84 , srv665028687:84 , srv686942390:84 , srv687369443:84 , srv722526257:84 , srv725590317:84 , srv778915376:84 , srv810492467:84 , srv812190229:84 , srv832106988:84 , srv832280363:84 , srv845406142:84 , srv926912964:84 , srv933398693:84 , srv2040323550:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-03-31 23:00:59,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-31 23:00:59,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv334263920=69, srv521375047=76, srv1014483938=0, srv1084073672=3, srv1969824445=50, srv1281656784=14, srv2079013959=62, srv615663105=82, srv659772160=85, srv236186232=65, srv1788087587=42, srv2040323550=57, srv1486002140=21, srv1239336532=10, srv1566761471=29, srv1931854191=47, srv34731578=70, srv1869218624=45, srv374079553=71, srv812190229=94, srv1642721394=35, srv2027271655=54, srv1527638091=27, srv1494769812=23, srv2118781429=64, srv810492467=93, srv933398693=99, srv1407047253=18, srv1920695380=46, srv832106988=95, srv1578972575=30, srv167487081=37, srv2106071889=63, srv316988670=68, srv194681324=48, srv1021506254=1, srv414481684=72, srv832280363=96, srv1782469508=40, srv1108796757=5, srv778915376=92, srv1363437273=16, srv687369443=89, srv65281114=84, srv1998571452=51, srv497634472=74, srv2032526871=55, srv2055432822=59, srv1516801510=25, srv1544015565=28, srv1411383544=19, srv451227853=73, srv1277114570=13, srv2026133758=53, srv60615455=81, srv1224185724=8, srv556269136=78, srv565166447=79, srv2036072612=56, srv1662821621=36, srv1522712963=26, srv1592112089=32, srv845406142=97, srv148267407=20, srv1832844274=44, srv926912964=98, srv1489382946=22, srv1641643155=34, srv1255005705=12, srv310979086=67, srv686942390=88, srv1807075525=43, srv1244401102=11, srv1784843618=41, srv725590317=91, srv137858070=17, srv722526257=90, srv262358735=66, srv1726376418=39, srv2022721097=52, srv2059711284=60, srv2063117797=61, srv498968558=75, srv665028687=87, srv66035524=86, srv623828435=83, srv1963205277=49, srv545380744=77, srv120302473=7, srv1495154121=24, srv2053665836=58, srv1313921885=15, srv1633378885=33, srv1162789374=6, srv1585280640=31, srv599062518=80, srv109052039=4, srv1722597674=38, srv1235397417=9, srv108070015=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-31 23:00:59,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-31 23:00:59,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-31 23:00:59,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-31 23:00:59,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-31 23:00:59,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-31 23:00:59,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-31 23:00:59,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-31 23:00:59,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-31 23:00:59,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-31 23:00:59,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-31 23:00:59,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-31 23:00:59,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-31 23:00:59,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-31 23:00:59,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-31 23:00:59,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-31 23:00:59,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-31 23:00:59,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-31 23:00:59,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-31 23:00:59,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-31 23:00:59,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-31 23:00:59,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-31 23:00:59,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-31 23:00:59,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-31 23:00:59,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-31 23:00:59,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-31 23:00:59,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-31 23:00:59,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-31 23:00:59,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-31 23:00:59,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-31 23:00:59,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-31 23:00:59,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-31 23:00:59,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-31 23:00:59,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-31 23:00:59,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-31 23:00:59,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-31 23:00:59,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-31 23:00:59,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-31 23:00:59,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-31 23:00:59,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-31 23:00:59,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-31 23:00:59,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-31 23:00:59,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-31 23:00:59,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-31 23:00:59,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-31 23:00:59,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-31 23:00:59,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-31 23:00:59,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-31 23:00:59,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-31 23:00:59,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-31 23:00:59,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-31 23:00:59,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-31 23:00:59,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-31 23:00:59,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-31 23:00:59,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-31 23:00:59,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-31 23:00:59,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-31 23:00:59,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-31 23:00:59,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-31 23:00:59,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-31 23:00:59,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-31 23:00:59,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-31 23:00:59,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-31 23:00:59,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-31 23:00:59,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-31 23:00:59,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-31 23:00:59,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-31 23:00:59,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-31 23:00:59,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-31 23:00:59,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-31 23:00:59,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-31 23:00:59,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-31 23:00:59,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-31 23:00:59,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-31 23:00:59,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-31 23:00:59,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-31 23:00:59,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-31 23:00:59,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-31 23:00:59,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-31 23:00:59,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-31 23:00:59,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-31 23:00:59,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-31 23:00:59,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-31 23:00:59,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-31 23:00:59,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-31 23:00:59,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-31 23:00:59,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-31 23:00:59,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-31 23:00:59,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-31 23:00:59,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-31 23:00:59,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-31 23:00:59,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-31 23:00:59,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-31 23:00:59,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-31 23:00:59,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-31 23:00:59,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-31 23:00:59,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-31 23:00:59,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-31 23:00:59,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-31 23:00:59,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-31 23:00:59,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-31 23:00:59,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-31 23:00:59,589 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 23:00:59,589 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-31 23:00:59,590 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-31 23:00:59,994 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 618 ms to try 5000 different iterations. Found a solution that moves 4671 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18405245054130218. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03982659758310724, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05477430480518337, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.519, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03982659758310724, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17933333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24583333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 23:01:00,008 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv615663105:81 , srv1782469508:82 , srv2059711284:82 , srv659772160:82 , srv1516801510:83 , srv1522712963:83 , srv1662821621:83 , srv194681324:83 , srv2036072612:83 , srv310979086:83 , srv414481684:83 , srv498968558:83 , srv60615455:83 , srv65281114:83 , srv109052039:84 , srv1235397417:84 , srv1363437273:84 , srv148267407:84 , srv1495154121:84 , srv1726376418:84 , srv2053665836:84 , srv2118781429:84 , srv262358735:84 , srv810492467:84 , srv108070015:85 , srv120302473:85 , srv1239336532:85 , srv1244401102:85 , srv1277114570:85 , srv1411383544:85 , srv1585280640:85 , srv1784843618:85 , srv1788087587:85 , srv34731578:85 , srv623828435:85 , srv66035524:85 , srv687369443:85 , srv832106988:85 , srv926912964:85 , srv933398693:85 , srv1084073672:86 , srv1641643155:86 , srv1722597674:86 , srv1963205277:86 , srv1969824445:86 , srv1998571452:86 , srv316988670:86 , srv556269136:86 , srv565166447:86 , srv599062518:86 , srv686942390:86 , srv722526257:86 , srv1313921885:87 , srv137858070:87 , srv1407047253:87 , srv1486002140:87 , srv1494769812:87 , srv1527638091:87 , srv1544015565:87 , srv1578972575:87 , srv2022721097:87 , srv2079013959:87 , srv374079553:87 , srv451227853:87 , srv725590317:87 , srv1021506254:88 , srv1807075525:88 , srv1832844274:88 , srv2026133758:88 , srv2055432822:88 , srv545380744:88 , srv832280363:88 , srv1014483938:89 , srv1108796757:89 , srv1162789374:89 , srv1255005705:89 , srv1281656784:89 , srv1489382946:89 , srv1592112089:89 , srv1642721394:89 , srv1869218624:89 , srv2032526871:89 , srv334263920:89 , srv497634472:89 , srv521375047:89 , srv845406142:89 , srv1224185724:90 , srv1920695380:90 , srv1931854191:90 , srv2027271655:90 , srv2063117797:90 , srv2106071889:90 , srv665028687:90 , srv778915376:90 , srv812190229:90 , srv1566761471:91 , srv1633378885:91 , srv167487081:91 , srv236186232:91 , srv2040323550:444 } 2023-03-31 23:01:00,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-31 23:01:00,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv334263920=69, srv521375047=76, srv1014483938=0, srv1084073672=3, srv1969824445=50, srv1281656784=14, srv2079013959=62, srv615663105=82, srv659772160=85, srv236186232=65, srv1788087587=42, srv2040323550=57, srv1486002140=21, srv1239336532=10, srv1566761471=29, srv1931854191=47, srv34731578=70, srv1869218624=45, srv374079553=71, srv812190229=94, srv1642721394=35, srv2027271655=54, srv1527638091=27, srv1494769812=23, srv2118781429=64, srv810492467=93, srv933398693=99, srv1407047253=18, srv1920695380=46, srv832106988=95, srv1578972575=30, srv167487081=37, srv2106071889=63, srv316988670=68, srv194681324=48, srv1021506254=1, srv414481684=72, srv832280363=96, srv1782469508=40, srv1108796757=5, srv778915376=92, srv1363437273=16, srv687369443=89, srv65281114=84, srv1998571452=51, srv497634472=74, srv2032526871=55, srv2055432822=59, srv1516801510=25, srv1544015565=28, srv1411383544=19, srv451227853=73, srv1277114570=13, srv2026133758=53, srv60615455=81, srv1224185724=8, srv556269136=78, srv565166447=79, srv2036072612=56, srv1662821621=36, srv1522712963=26, srv1592112089=32, srv845406142=97, srv148267407=20, srv1832844274=44, srv926912964=98, srv1489382946=22, srv1641643155=34, srv1255005705=12, srv310979086=67, srv686942390=88, srv1807075525=43, srv1244401102=11, srv1784843618=41, srv725590317=91, srv137858070=17, srv722526257=90, srv262358735=66, srv1726376418=39, srv2022721097=52, srv2059711284=60, srv2063117797=61, srv498968558=75, srv665028687=87, srv66035524=86, srv623828435=83, srv1963205277=49, srv545380744=77, srv120302473=7, srv1495154121=24, srv2053665836=58, srv1313921885=15, srv1633378885=33, srv1162789374=6, srv1585280640=31, srv599062518=80, srv109052039=4, srv1722597674=38, srv1235397417=9, srv108070015=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-31 23:01:00,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-31 23:01:00,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-31 23:01:00,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-31 23:01:00,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-31 23:01:00,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-31 23:01:00,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-31 23:01:00,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-31 23:01:00,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-31 23:01:00,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-31 23:01:00,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-31 23:01:00,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-31 23:01:00,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-31 23:01:00,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-31 23:01:00,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-31 23:01:00,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-31 23:01:00,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-31 23:01:00,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-31 23:01:00,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-31 23:01:00,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-31 23:01:00,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-31 23:01:00,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-31 23:01:00,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-31 23:01:00,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-31 23:01:00,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-31 23:01:00,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-31 23:01:00,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-31 23:01:00,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-31 23:01:00,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-31 23:01:00,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-31 23:01:00,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-31 23:01:00,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-31 23:01:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-31 23:01:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-31 23:01:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-31 23:01:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-31 23:01:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-31 23:01:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-31 23:01:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-31 23:01:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-31 23:01:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-31 23:01:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-31 23:01:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-31 23:01:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-31 23:01:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-31 23:01:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-31 23:01:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-31 23:01:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-31 23:01:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-31 23:01:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-31 23:01:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-31 23:01:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-31 23:01:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-31 23:01:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-31 23:01:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-31 23:01:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-31 23:01:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-31 23:01:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-31 23:01:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-31 23:01:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-31 23:01:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-31 23:01:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-31 23:01:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-31 23:01:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-31 23:01:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-31 23:01:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-31 23:01:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-31 23:01:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-31 23:01:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-31 23:01:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-31 23:01:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-31 23:01:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-31 23:01:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-31 23:01:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-31 23:01:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-31 23:01:00,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-31 23:01:00,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-31 23:01:00,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-31 23:01:00,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-31 23:01:00,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-31 23:01:00,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-31 23:01:00,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-31 23:01:00,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-31 23:01:00,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-31 23:01:00,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-31 23:01:00,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-31 23:01:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-31 23:01:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-31 23:01:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-31 23:01:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-31 23:01:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-31 23:01:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-31 23:01:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-31 23:01:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-31 23:01:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-31 23:01:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-31 23:01:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-31 23:01:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-31 23:01:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-31 23:01:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-31 23:01:00,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-31 23:01:00,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-31 23:01:00,185 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03982659758310724, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05477430480518337, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03982659758310724, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17933333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24583333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 23:01:00,186 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-31 23:01:00,186 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18401974350024058, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03982659758310724, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05477430480518337, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03982659758310724, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17933333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24583333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-31 23:01:00,369 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 299 ms to try 5000 different iterations. Found a solution that moves 2611 regions; Going from a computed imbalance of 0.18401974350024058 to a new imbalance of 0.01611870154974359. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019814578808500426); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.033904202534419677, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2901111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019814578808500426); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04858333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 23:01:00,377 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1633378885:80 , srv1920695380:81 , srv1255005705:82 , srv1281656784:82 , srv1489382946:82 , srv1566761471:82 , srv1592112089:82 , srv167487081:82 , srv1869218624:82 , srv2063117797:82 , srv236186232:82 , srv497634472:82 , srv521375047:82 , srv812190229:82 , srv1108796757:83 , srv1407047253:83 , srv1578972575:83 , srv1642721394:83 , srv1832844274:83 , srv2027271655:83 , srv2032526871:83 , srv2106071889:83 , srv665028687:83 , srv845406142:83 , srv1014483938:84 , srv1162789374:84 , srv1313921885:84 , srv1641643155:84 , srv1931854191:84 , srv2026133758:84 , srv2055432822:84 , srv334263920:84 , srv565166447:84 , srv778915376:84 , srv1021506254:85 , srv1224185724:85 , srv137858070:85 , srv1494769812:85 , srv1807075525:85 , srv1963205277:85 , srv1998571452:85 , srv545380744:85 , srv832280363:85 , srv926912964:85 , srv108070015:86 , srv1084073672:86 , srv109052039:86 , srv1486002140:86 , srv1522712963:86 , srv1527638091:86 , srv1544015565:86 , srv1722597674:86 , srv1969824445:86 , srv2079013959:86 , srv2118781429:86 , srv316988670:86 , srv34731578:86 , srv374079553:86 , srv451227853:86 , srv722526257:86 , srv832106988:86 , srv120302473:87 , srv1235397417:87 , srv148267407:87 , srv1585280640:87 , srv1788087587:87 , srv262358735:87 , srv65281114:87 , srv66035524:87 , srv686942390:87 , srv933398693:87 , srv1726376418:88 , srv194681324:88 , srv2022721097:88 , srv2053665836:88 , srv414481684:88 , srv623828435:88 , srv725590317:88 , srv810492467:88 , srv1239336532:89 , srv1244401102:89 , srv1277114570:89 , srv1363437273:89 , srv1411383544:89 , srv1516801510:89 , srv1784843618:89 , srv2036072612:89 , srv310979086:89 , srv498968558:89 , srv599062518:89 , srv615663105:89 , srv687369443:89 , srv1662821621:90 , srv1782469508:90 , srv556269136:90 , srv60615455:90 , srv2059711284:91 , srv1495154121:92 , srv659772160:93 , srv2040323550:506 } 2023-03-31 23:01:00,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-31 23:01:00,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv334263920=69, srv521375047=76, srv1014483938=0, srv1084073672=3, srv1969824445=50, srv1281656784=14, srv2079013959=62, srv615663105=82, srv659772160=85, srv236186232=65, srv1788087587=42, srv2040323550=57, srv1486002140=21, srv1239336532=10, srv1566761471=29, srv1931854191=47, srv34731578=70, srv1869218624=45, srv374079553=71, srv812190229=94, srv1642721394=35, srv2027271655=54, srv1527638091=27, srv1494769812=23, srv2118781429=64, srv810492467=93, srv933398693=99, srv1407047253=18, srv1920695380=46, srv832106988=95, srv1578972575=30, srv167487081=37, srv2106071889=63, srv316988670=68, srv194681324=48, srv1021506254=1, srv414481684=72, srv832280363=96, srv1782469508=40, srv1108796757=5, srv778915376=92, srv1363437273=16, srv687369443=89, srv65281114=84, srv1998571452=51, srv497634472=74, srv2032526871=55, srv2055432822=59, srv1516801510=25, srv1544015565=28, srv1411383544=19, srv451227853=73, srv1277114570=13, srv2026133758=53, srv60615455=81, srv1224185724=8, srv556269136=78, srv565166447=79, srv2036072612=56, srv1662821621=36, srv1522712963=26, srv1592112089=32, srv845406142=97, srv148267407=20, srv1832844274=44, srv926912964=98, srv1489382946=22, srv1641643155=34, srv1255005705=12, srv310979086=67, srv686942390=88, srv1807075525=43, srv1244401102=11, srv1784843618=41, srv725590317=91, srv137858070=17, srv722526257=90, srv262358735=66, srv1726376418=39, srv2022721097=52, srv2059711284=60, srv2063117797=61, srv498968558=75, srv665028687=87, srv66035524=86, srv623828435=83, srv1963205277=49, srv545380744=77, srv120302473=7, srv1495154121=24, srv2053665836=58, srv1313921885=15, srv1633378885=33, srv1162789374=6, srv1585280640=31, srv599062518=80, srv109052039=4, srv1722597674=38, srv1235397417=9, srv108070015=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-31 23:01:00,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-31 23:01:00,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-31 23:01:00,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-31 23:01:00,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-31 23:01:00,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-31 23:01:00,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-31 23:01:00,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-31 23:01:00,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-31 23:01:00,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-31 23:01:00,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-31 23:01:00,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-31 23:01:00,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-31 23:01:00,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-31 23:01:00,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-31 23:01:00,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-31 23:01:00,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-31 23:01:00,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-31 23:01:00,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-31 23:01:00,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-31 23:01:00,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-31 23:01:00,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-31 23:01:00,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-31 23:01:00,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-31 23:01:00,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-31 23:01:00,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-31 23:01:00,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-31 23:01:00,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-31 23:01:00,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-31 23:01:00,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-31 23:01:00,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-31 23:01:00,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-31 23:01:00,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-31 23:01:00,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-31 23:01:00,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-31 23:01:00,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-31 23:01:00,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-31 23:01:00,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-31 23:01:00,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-31 23:01:00,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-31 23:01:00,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-31 23:01:00,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-31 23:01:00,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-31 23:01:00,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-31 23:01:00,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-31 23:01:00,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-31 23:01:00,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-31 23:01:00,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-31 23:01:00,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-31 23:01:00,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-31 23:01:00,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-31 23:01:00,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-31 23:01:00,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-31 23:01:00,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-31 23:01:00,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-31 23:01:00,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-31 23:01:00,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-31 23:01:00,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-31 23:01:00,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-31 23:01:00,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-31 23:01:00,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-31 23:01:00,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-31 23:01:00,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-31 23:01:00,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-31 23:01:00,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-31 23:01:00,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-31 23:01:00,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-31 23:01:00,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-31 23:01:00,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-31 23:01:00,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-31 23:01:00,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-31 23:01:00,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-31 23:01:00,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-31 23:01:00,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-31 23:01:00,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-31 23:01:00,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-31 23:01:00,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-31 23:01:00,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-31 23:01:00,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-31 23:01:00,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-31 23:01:00,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-31 23:01:00,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-31 23:01:00,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-31 23:01:00,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-31 23:01:00,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-31 23:01:00,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-31 23:01:00,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-31 23:01:00,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-31 23:01:00,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-31 23:01:00,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-31 23:01:00,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-31 23:01:00,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-31 23:01:00,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-31 23:01:00,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-31 23:01:00,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-31 23:01:00,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-31 23:01:00,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-31 23:01:00,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-31 23:01:00,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-31 23:01:00,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-31 23:01:00,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-31 23:01:00,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-31 23:01:00,497 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019814578808500426); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.033904202534419677, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019814578808500426); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04858333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 23:01:00,498 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-31 23:01:00,498 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01610041893698147, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019814578808500426); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.033904202534419677, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019814578808500426); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04858333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-31 23:01:00,660 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 242 ms to try 5000 different iterations. Found a solution that moves 1075 regions; Going from a computed imbalance of 0.01610041893698147 to a new imbalance of 6.709104539275575E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004877857158646318); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007153911599848797); MoveCostFunction : (multiplier=7.0, imbalance=0.11944444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004877857158646318); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 23:01:00,662 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1224185724:82 , srv1495154121:82 , srv2022721097:82 , srv556269136:82 , srv599062518:82 , srv1239336532:83 , srv1411383544:83 , srv1784843618:83 , srv659772160:83 , srv1014483938:84 , srv1021506254:84 , srv1162789374:84 , srv1235397417:84 , srv1244401102:84 , srv1277114570:84 , srv1363437273:84 , srv1566761471:84 , srv2079013959:84 , srv374079553:84 , srv686942390:84 , srv687369443:84 , srv725590317:84 , srv778915376:84 , srv832280363:84 , srv1084073672:85 , srv1108796757:85 , srv1313921885:85 , srv137858070:85 , srv1486002140:85 , srv1527638091:85 , srv1544015565:85 , srv1585280640:85 , srv1662821621:85 , srv167487081:85 , srv1722597674:85 , srv1726376418:85 , srv1788087587:85 , srv1807075525:85 , srv1931854191:85 , srv2027271655:85 , srv2059711284:85 , srv2106071889:85 , srv236186232:85 , srv310979086:85 , srv316988670:85 , srv451227853:85 , srv498968558:85 , srv545380744:85 , srv60615455:85 , srv623828435:85 , srv665028687:85 , srv810492467:85 , srv933398693:85 , srv120302473:86 , srv148267407:86 , srv1489382946:86 , srv1494769812:86 , srv1516801510:86 , srv1592112089:86 , srv1642721394:86 , srv1782469508:86 , srv1869218624:86 , srv194681324:86 , srv1969824445:86 , srv2026133758:86 , srv2036072612:86 , srv2053665836:86 , srv2055432822:86 , srv2063117797:86 , srv262358735:86 , srv334263920:86 , srv34731578:86 , srv66035524:86 , srv722526257:86 , srv812190229:86 , srv845406142:86 , srv108070015:87 , srv109052039:87 , srv1255005705:87 , srv1281656784:87 , srv1407047253:87 , srv1578972575:87 , srv1633378885:87 , srv1641643155:87 , srv1832844274:87 , srv1920695380:87 , srv1963205277:87 , srv1998571452:87 , srv2032526871:87 , srv414481684:87 , srv497634472:87 , srv521375047:87 , srv615663105:87 , srv65281114:87 , srv832106988:87 , srv926912964:87 , srv1522712963:88 , srv2118781429:88 , srv565166447:88 , srv2040323550:551 } 2023-03-31 23:01:00,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-31 23:01:00,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv334263920=69, srv521375047=76, srv1014483938=0, srv1084073672=3, srv1969824445=50, srv1281656784=14, srv2079013959=62, srv615663105=82, srv659772160=85, srv236186232=65, srv1788087587=42, srv2040323550=57, srv1486002140=21, srv1239336532=10, srv1566761471=29, srv1931854191=47, srv34731578=70, srv1869218624=45, srv374079553=71, srv812190229=94, srv1642721394=35, srv2027271655=54, srv1527638091=27, srv1494769812=23, srv2118781429=64, srv810492467=93, srv933398693=99, srv1407047253=18, srv1920695380=46, srv832106988=95, srv1578972575=30, srv167487081=37, srv2106071889=63, srv316988670=68, srv194681324=48, srv1021506254=1, srv414481684=72, srv832280363=96, srv1782469508=40, srv1108796757=5, srv778915376=92, srv1363437273=16, srv687369443=89, srv65281114=84, srv1998571452=51, srv497634472=74, srv2032526871=55, srv2055432822=59, srv1516801510=25, srv1544015565=28, srv1411383544=19, srv451227853=73, srv1277114570=13, srv2026133758=53, srv60615455=81, srv1224185724=8, srv556269136=78, srv565166447=79, srv2036072612=56, srv1662821621=36, srv1522712963=26, srv1592112089=32, srv845406142=97, srv148267407=20, srv1832844274=44, srv926912964=98, srv1489382946=22, srv1641643155=34, srv1255005705=12, srv310979086=67, srv686942390=88, srv1807075525=43, srv1244401102=11, srv1784843618=41, srv725590317=91, srv137858070=17, srv722526257=90, srv262358735=66, srv1726376418=39, srv2022721097=52, srv2059711284=60, srv2063117797=61, srv498968558=75, srv665028687=87, srv66035524=86, srv623828435=83, srv1963205277=49, srv545380744=77, srv120302473=7, srv1495154121=24, srv2053665836=58, srv1313921885=15, srv1633378885=33, srv1162789374=6, srv1585280640=31, srv599062518=80, srv109052039=4, srv1722597674=38, srv1235397417=9, srv108070015=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-31 23:01:00,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-31 23:01:00,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-31 23:01:00,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-31 23:01:00,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-31 23:01:00,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-31 23:01:00,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-31 23:01:00,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-31 23:01:00,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-31 23:01:00,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-31 23:01:00,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-31 23:01:00,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-31 23:01:00,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-31 23:01:00,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-31 23:01:00,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-31 23:01:00,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-31 23:01:00,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-31 23:01:00,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-31 23:01:00,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-31 23:01:00,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-31 23:01:00,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-31 23:01:00,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-31 23:01:00,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-31 23:01:00,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-31 23:01:00,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-31 23:01:00,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-31 23:01:00,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-31 23:01:00,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-31 23:01:00,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-31 23:01:00,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-31 23:01:00,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-31 23:01:00,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-31 23:01:00,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-31 23:01:00,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-31 23:01:00,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-31 23:01:00,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-31 23:01:00,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-31 23:01:00,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-31 23:01:00,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-31 23:01:00,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-31 23:01:00,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-31 23:01:00,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-31 23:01:00,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-31 23:01:00,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-31 23:01:00,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-31 23:01:00,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-31 23:01:00,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-31 23:01:00,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-31 23:01:00,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-31 23:01:00,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-31 23:01:00,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-31 23:01:00,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-31 23:01:00,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-31 23:01:00,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-31 23:01:00,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-31 23:01:00,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-31 23:01:00,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-31 23:01:00,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-31 23:01:00,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-31 23:01:00,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-31 23:01:00,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-31 23:01:00,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-31 23:01:00,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-31 23:01:00,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-31 23:01:00,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-31 23:01:00,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-31 23:01:00,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-31 23:01:00,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-31 23:01:00,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-31 23:01:00,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-31 23:01:00,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-31 23:01:00,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-31 23:01:00,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-31 23:01:00,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-31 23:01:00,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-31 23:01:00,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-31 23:01:00,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-31 23:01:00,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-31 23:01:00,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-31 23:01:00,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-31 23:01:00,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-31 23:01:00,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-31 23:01:00,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-31 23:01:00,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-31 23:01:00,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-31 23:01:00,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-31 23:01:00,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-31 23:01:00,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-31 23:01:00,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-31 23:01:00,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-31 23:01:00,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-31 23:01:00,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-31 23:01:00,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-31 23:01:00,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-31 23:01:00,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-31 23:01:00,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-31 23:01:00,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-31 23:01:00,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-31 23:01:00,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-31 23:01:00,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-31 23:01:00,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-31 23:01:00,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-31 23:01:00,799 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004877857158646318); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007153911599848797); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004877857158646318); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 23:01:00,799 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-31 23:01:00,799 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.633831430431158E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004877857158646318); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007153911599848797); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004877857158646318); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-31 23:01:01,014 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 313 ms to try 5000 different iterations. Found a solution that moves 242 regions; Going from a computed imbalance of 6.633831430431158E-4 to a new imbalance of 2.775973390915972E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.02688888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 23:01:01,015 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1486002140:83 , srv1495154121:83 , srv1633378885:83 , srv1931854191:83 , srv1963205277:83 , srv2032526871:83 , srv334263920:83 , srv725590317:83 , srv845406142:83 , srv108070015:84 , srv120302473:84 , srv1239336532:84 , srv1244401102:84 , srv1255005705:84 , srv1277114570:84 , srv1281656784:84 , srv137858070:84 , srv1494769812:84 , srv1516801510:84 , srv1544015565:84 , srv1662821621:84 , srv167487081:84 , srv1782469508:84 , srv1807075525:84 , srv1832844274:84 , srv1920695380:84 , srv1969824445:84 , srv1998571452:84 , srv2026133758:84 , srv2036072612:84 , srv2053665836:84 , srv2055432822:84 , srv2059711284:84 , srv2063117797:84 , srv2106071889:84 , srv2118781429:84 , srv236186232:84 , srv414481684:84 , srv451227853:84 , srv497634472:84 , srv521375047:84 , srv545380744:84 , srv556269136:84 , srv565166447:84 , srv60615455:84 , srv623828435:84 , srv66035524:84 , srv687369443:84 , srv722526257:84 , srv778915376:84 , srv812190229:84 , srv832106988:84 , srv1014483938:85 , srv1021506254:85 , srv1084073672:85 , srv109052039:85 , srv1108796757:85 , srv1162789374:85 , srv1224185724:85 , srv1363437273:85 , srv1407047253:85 , srv1411383544:85 , srv148267407:85 , srv1489382946:85 , srv1522712963:85 , srv1527638091:85 , srv1566761471:85 , srv1578972575:85 , srv1592112089:85 , srv1641643155:85 , srv1642721394:85 , srv1722597674:85 , srv1726376418:85 , srv1784843618:85 , srv1788087587:85 , srv1869218624:85 , srv194681324:85 , srv2022721097:85 , srv2027271655:85 , srv2079013959:85 , srv262358735:85 , srv310979086:85 , srv316988670:85 , srv34731578:85 , srv374079553:85 , srv498968558:85 , srv599062518:85 , srv65281114:85 , srv659772160:85 , srv665028687:85 , srv686942390:85 , srv810492467:85 , srv832280363:85 , srv926912964:85 , srv933398693:85 , srv1313921885:86 , srv1585280640:86 , srv615663105:86 , srv1235397417:87 , srv2040323550:641 } 2023-03-31 23:01:01,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-31 23:01:01,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv334263920=69, srv521375047=76, srv1014483938=0, srv1084073672=3, srv1969824445=50, srv1281656784=14, srv2079013959=62, srv615663105=82, srv659772160=85, srv236186232=65, srv1788087587=42, srv2040323550=57, srv1486002140=21, srv1239336532=10, srv1566761471=29, srv1931854191=47, srv34731578=70, srv1869218624=45, srv374079553=71, srv812190229=94, srv1642721394=35, srv2027271655=54, srv1527638091=27, srv1494769812=23, srv2118781429=64, srv810492467=93, srv933398693=99, srv1407047253=18, srv1920695380=46, srv832106988=95, srv1578972575=30, srv167487081=37, srv2106071889=63, srv316988670=68, srv194681324=48, srv1021506254=1, srv414481684=72, srv832280363=96, srv1782469508=40, srv1108796757=5, srv778915376=92, srv1363437273=16, srv687369443=89, srv65281114=84, srv1998571452=51, srv497634472=74, srv2032526871=55, srv2055432822=59, srv1516801510=25, srv1544015565=28, srv1411383544=19, srv451227853=73, srv1277114570=13, srv2026133758=53, srv60615455=81, srv1224185724=8, srv556269136=78, srv565166447=79, srv2036072612=56, srv1662821621=36, srv1522712963=26, srv1592112089=32, srv845406142=97, srv148267407=20, srv1832844274=44, srv926912964=98, srv1489382946=22, srv1641643155=34, srv1255005705=12, srv310979086=67, srv686942390=88, srv1807075525=43, srv1244401102=11, srv1784843618=41, srv725590317=91, srv137858070=17, srv722526257=90, srv262358735=66, srv1726376418=39, srv2022721097=52, srv2059711284=60, srv2063117797=61, srv498968558=75, srv665028687=87, srv66035524=86, srv623828435=83, srv1963205277=49, srv545380744=77, srv120302473=7, srv1495154121=24, srv2053665836=58, srv1313921885=15, srv1633378885=33, srv1162789374=6, srv1585280640=31, srv599062518=80, srv109052039=4, srv1722597674=38, srv1235397417=9, srv108070015=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-31 23:01:01,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-31 23:01:01,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-31 23:01:01,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-31 23:01:01,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-31 23:01:01,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-31 23:01:01,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-31 23:01:01,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-31 23:01:01,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-31 23:01:01,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-31 23:01:01,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-31 23:01:01,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-31 23:01:01,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-31 23:01:01,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-31 23:01:01,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-31 23:01:01,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-31 23:01:01,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-31 23:01:01,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-31 23:01:01,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-31 23:01:01,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-31 23:01:01,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-31 23:01:01,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-31 23:01:01,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-31 23:01:01,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-31 23:01:01,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-31 23:01:01,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-31 23:01:01,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-31 23:01:01,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-31 23:01:01,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-31 23:01:01,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-31 23:01:01,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-31 23:01:01,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-31 23:01:01,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-31 23:01:01,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-31 23:01:01,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-31 23:01:01,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-31 23:01:01,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-31 23:01:01,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-31 23:01:01,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-31 23:01:01,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-31 23:01:01,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-31 23:01:01,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-31 23:01:01,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-31 23:01:01,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-31 23:01:01,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-31 23:01:01,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-31 23:01:01,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-31 23:01:01,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-31 23:01:01,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-31 23:01:01,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-31 23:01:01,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-31 23:01:01,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-31 23:01:01,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-31 23:01:01,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-31 23:01:01,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-31 23:01:01,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-31 23:01:01,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-31 23:01:01,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-31 23:01:01,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-31 23:01:01,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-31 23:01:01,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-31 23:01:01,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-31 23:01:01,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-31 23:01:01,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-31 23:01:01,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-31 23:01:01,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-31 23:01:01,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-31 23:01:01,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-31 23:01:01,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-31 23:01:01,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-31 23:01:01,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-31 23:01:01,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-31 23:01:01,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-31 23:01:01,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-31 23:01:01,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-31 23:01:01,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-31 23:01:01,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-31 23:01:01,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-31 23:01:01,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-31 23:01:01,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-31 23:01:01,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-31 23:01:01,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-31 23:01:01,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-31 23:01:01,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-31 23:01:01,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-31 23:01:01,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-31 23:01:01,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-31 23:01:01,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-31 23:01:01,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-31 23:01:01,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-31 23:01:01,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-31 23:01:01,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-31 23:01:01,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-31 23:01:01,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-31 23:01:01,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-31 23:01:01,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-31 23:01:01,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-31 23:01:01,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-31 23:01:01,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-31 23:01:01,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-31 23:01:01,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-31 23:01:01,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-31 23:01:01,118 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 23:01:01,118 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-31 23:01:01,119 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.6065213691452885E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-31 23:01:01,278 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 238 ms to try 5000 different iterations. Found a solution that moves 30 regions; Going from a computed imbalance of 2.6065213691452885E-5 to a new imbalance of 6.242143229526737E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0013400504203456161); MoveCostFunction : (multiplier=7.0, imbalance=0.0033333333333333335); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 23:01:01,279 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1527638091:83 , srv1585280640:83 , srv1642721394:83 , srv2027271655:83 , srv615663105:83 , srv659772160:83 , srv665028687:83 , srv1014483938:84 , srv1021506254:84 , srv108070015:84 , srv1084073672:84 , srv109052039:84 , srv1108796757:84 , srv1162789374:84 , srv120302473:84 , srv1224185724:84 , srv1235397417:84 , srv1244401102:84 , srv1255005705:84 , srv1277114570:84 , srv1281656784:84 , srv1313921885:84 , srv1363437273:84 , srv1407047253:84 , srv1411383544:84 , srv148267407:84 , srv1489382946:84 , srv1494769812:84 , srv1516801510:84 , srv1522712963:84 , srv1544015565:84 , srv1566761471:84 , srv1578972575:84 , srv1592112089:84 , srv1641643155:84 , srv1662821621:84 , srv167487081:84 , srv1722597674:84 , srv1726376418:84 , srv1782469508:84 , srv1784843618:84 , srv1788087587:84 , srv1807075525:84 , srv1832844274:84 , srv1869218624:84 , srv1920695380:84 , srv1931854191:84 , srv194681324:84 , srv1969824445:84 , srv1998571452:84 , srv2022721097:84 , srv2026133758:84 , srv2032526871:84 , srv2036072612:84 , srv2053665836:84 , srv2055432822:84 , srv2059711284:84 , srv2063117797:84 , srv2079013959:84 , srv2106071889:84 , srv2118781429:84 , srv236186232:84 , srv262358735:84 , srv310979086:84 , srv316988670:84 , srv334263920:84 , srv34731578:84 , srv374079553:84 , srv414481684:84 , srv451227853:84 , srv497634472:84 , srv498968558:84 , srv521375047:84 , srv545380744:84 , srv556269136:84 , srv565166447:84 , srv599062518:84 , srv60615455:84 , srv623828435:84 , srv65281114:84 , srv66035524:84 , srv686942390:84 , srv687369443:84 , srv722526257:84 , srv725590317:84 , srv778915376:84 , srv810492467:84 , srv812190229:84 , srv832106988:84 , srv832280363:84 , srv926912964:84 , srv933398693:84 , srv1239336532:85 , srv137858070:85 , srv1486002140:85 , srv1495154121:85 , srv1633378885:85 , srv1963205277:85 , srv845406142:85 , srv2040323550:684 } 2023-03-31 23:01:01,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-31 23:01:01,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv334263920=69, srv521375047=76, srv1014483938=0, srv1084073672=3, srv1969824445=50, srv1281656784=14, srv2079013959=62, srv615663105=82, srv659772160=85, srv236186232=65, srv1788087587=42, srv2040323550=57, srv1486002140=21, srv1239336532=10, srv1566761471=29, srv1931854191=47, srv34731578=70, srv1869218624=45, srv374079553=71, srv812190229=94, srv1642721394=35, srv2027271655=54, srv1527638091=27, srv1494769812=23, srv2118781429=64, srv810492467=93, srv933398693=99, srv1407047253=18, srv1920695380=46, srv832106988=95, srv1578972575=30, srv167487081=37, srv2106071889=63, srv316988670=68, srv194681324=48, srv1021506254=1, srv414481684=72, srv832280363=96, srv1782469508=40, srv1108796757=5, srv778915376=92, srv1363437273=16, srv687369443=89, srv65281114=84, srv1998571452=51, srv497634472=74, srv2032526871=55, srv2055432822=59, srv1516801510=25, srv1544015565=28, srv1411383544=19, srv451227853=73, srv1277114570=13, srv2026133758=53, srv60615455=81, srv1224185724=8, srv556269136=78, srv565166447=79, srv2036072612=56, srv1662821621=36, srv1522712963=26, srv1592112089=32, srv845406142=97, srv148267407=20, srv1832844274=44, srv926912964=98, srv1489382946=22, srv1641643155=34, srv1255005705=12, srv310979086=67, srv686942390=88, srv1807075525=43, srv1244401102=11, srv1784843618=41, srv725590317=91, srv137858070=17, srv722526257=90, srv262358735=66, srv1726376418=39, srv2022721097=52, srv2059711284=60, srv2063117797=61, srv498968558=75, srv665028687=87, srv66035524=86, srv623828435=83, srv1963205277=49, srv545380744=77, srv120302473=7, srv1495154121=24, srv2053665836=58, srv1313921885=15, srv1633378885=33, srv1162789374=6, srv1585280640=31, srv599062518=80, srv109052039=4, srv1722597674=38, srv1235397417=9, srv108070015=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-31 23:01:01,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-31 23:01:01,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-31 23:01:01,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-31 23:01:01,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-31 23:01:01,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-31 23:01:01,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-31 23:01:01,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-31 23:01:01,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-31 23:01:01,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-31 23:01:01,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-31 23:01:01,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-31 23:01:01,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-31 23:01:01,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-31 23:01:01,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-31 23:01:01,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-31 23:01:01,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-31 23:01:01,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-31 23:01:01,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-31 23:01:01,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-31 23:01:01,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-31 23:01:01,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-31 23:01:01,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-31 23:01:01,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-31 23:01:01,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-31 23:01:01,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-31 23:01:01,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-31 23:01:01,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-31 23:01:01,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-31 23:01:01,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-31 23:01:01,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-31 23:01:01,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-31 23:01:01,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-31 23:01:01,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-31 23:01:01,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-31 23:01:01,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-31 23:01:01,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-31 23:01:01,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-31 23:01:01,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-31 23:01:01,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-31 23:01:01,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-31 23:01:01,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-31 23:01:01,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-31 23:01:01,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-31 23:01:01,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-31 23:01:01,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-31 23:01:01,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-31 23:01:01,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-31 23:01:01,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-31 23:01:01,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-31 23:01:01,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-31 23:01:01,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-31 23:01:01,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-31 23:01:01,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-31 23:01:01,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-31 23:01:01,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-31 23:01:01,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-31 23:01:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-31 23:01:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-31 23:01:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-31 23:01:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-31 23:01:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-31 23:01:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-31 23:01:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-31 23:01:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-31 23:01:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-31 23:01:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-31 23:01:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-31 23:01:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-31 23:01:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-31 23:01:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-31 23:01:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-31 23:01:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-31 23:01:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-31 23:01:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-31 23:01:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-31 23:01:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-31 23:01:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-31 23:01:01,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-31 23:01:01,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-31 23:01:01,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-31 23:01:01,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-31 23:01:01,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-31 23:01:01,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-31 23:01:01,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-31 23:01:01,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-31 23:01:01,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-31 23:01:01,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-31 23:01:01,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-31 23:01:01,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-31 23:01:01,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-31 23:01:01,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-31 23:01:01,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-31 23:01:01,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-31 23:01:01,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-31 23:01:01,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-31 23:01:01,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-31 23:01:01,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-31 23:01:01,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-31 23:01:01,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-31 23:01:01,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-31 23:01:01,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-31 23:01:01,381 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.032078739728369E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0013400504203456161); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 23:01:01,381 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1527638091:83 , srv1585280640:83 , srv1642721394:83 , srv2027271655:83 , srv615663105:83 , srv659772160:83 , srv665028687:83 , srv1014483938:84 , srv1021506254:84 , srv108070015:84 , srv1084073672:84 , srv109052039:84 , srv1108796757:84 , srv1162789374:84 , srv120302473:84 , srv1224185724:84 , srv1235397417:84 , srv1244401102:84 , srv1255005705:84 , srv1277114570:84 , srv1281656784:84 , srv1313921885:84 , srv1363437273:84 , srv1407047253:84 , srv1411383544:84 , srv148267407:84 , srv1489382946:84 , srv1494769812:84 , srv1516801510:84 , srv1522712963:84 , srv1544015565:84 , srv1566761471:84 , srv1578972575:84 , srv1592112089:84 , srv1641643155:84 , srv1662821621:84 , srv167487081:84 , srv1722597674:84 , srv1726376418:84 , srv1782469508:84 , srv1784843618:84 , srv1788087587:84 , srv1807075525:84 , srv1832844274:84 , srv1869218624:84 , srv1920695380:84 , srv1931854191:84 , srv194681324:84 , srv1969824445:84 , srv1998571452:84 , srv2022721097:84 , srv2026133758:84 , srv2032526871:84 , srv2036072612:84 , srv2053665836:84 , srv2055432822:84 , srv2059711284:84 , srv2063117797:84 , srv2079013959:84 , srv2106071889:84 , srv2118781429:84 , srv236186232:84 , srv262358735:84 , srv310979086:84 , srv316988670:84 , srv334263920:84 , srv34731578:84 , srv374079553:84 , srv414481684:84 , srv451227853:84 , srv497634472:84 , srv498968558:84 , srv521375047:84 , srv545380744:84 , srv556269136:84 , srv565166447:84 , srv599062518:84 , srv60615455:84 , srv623828435:84 , srv65281114:84 , srv66035524:84 , srv686942390:84 , srv687369443:84 , srv722526257:84 , srv725590317:84 , srv778915376:84 , srv810492467:84 , srv812190229:84 , srv832106988:84 , srv832280363:84 , srv926912964:84 , srv933398693:84 , srv1239336532:85 , srv137858070:85 , srv1486002140:85 , srv1495154121:85 , srv1633378885:85 , srv1963205277:85 , srv845406142:85 , srv2040323550:684 } 2023-03-31 23:01:01,403 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=280 (was 280), ProcessCount=164 (was 164), AvailableMemoryMB=8898 (was 9398) 2023-03-31 23:01:01,411 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=280, ProcessCount=164, AvailableMemoryMB=8898 2023-03-31 23:01:01,412 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-31 23:01:01,412 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-03-31 23:01:01,424 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1105347588:84 , srv1152293358:84 , srv1163855510:84 , srv1181170312:84 , srv1209382876:84 , srv1373901155:84 , srv1444630992:84 , srv1587468722:84 , srv1618131291:84 , srv1631318319:84 , srv1646718812:84 , srv1692718046:84 , srv2004045914:84 , srv2015476579:84 , srv2054721217:84 , srv2089304497:84 , srv24323774:84 , srv268246896:84 , srv336783785:84 , srv355265103:84 , srv409702318:84 , srv445318292:84 , srv53282439:84 , srv539436965:84 , srv614284600:84 , srv731027329:84 , srv823221781:84 , srv937162167:84 , srv953623969:84 , srv671815061:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-03-31 23:01:01,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-31 23:01:01,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv731027329=26, srv614284600=24, srv1181170312=3, srv24323774=16, srv1618131291=8, srv1646718812=10, srv409702318=20, srv2015476579=13, srv355265103=19, srv1152293358=1, srv953623969=29, srv336783785=18, srv2054721217=14, srv1209382876=4, srv937162167=28, srv1444630992=6, srv1631318319=9, srv1373901155=5, srv2089304497=15, srv1692718046=11, srv539436965=23, srv1163855510=2, srv2004045914=12, srv445318292=21, srv1105347588=0, srv671815061=25, srv53282439=22, srv1587468722=7, srv823221781=27, srv268246896=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-31 23:01:01,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-31 23:01:01,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-31 23:01:01,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-31 23:01:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-31 23:01:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-31 23:01:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-31 23:01:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-31 23:01:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-31 23:01:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-31 23:01:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-31 23:01:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-31 23:01:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-31 23:01:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-31 23:01:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-31 23:01:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-31 23:01:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-31 23:01:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-31 23:01:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-31 23:01:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-31 23:01:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-31 23:01:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-31 23:01:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-31 23:01:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-31 23:01:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-31 23:01:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-31 23:01:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-31 23:01:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-31 23:01:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-31 23:01:01,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-31 23:01:01,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-31 23:01:01,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-03-31 23:01:01,467 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1168983005805906, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 23:01:01,468 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912561385992329, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1168983005805906, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-03-31 23:03:01,447 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9807958 different iterations. Found a solution that moves 1833 regions; Going from a computed imbalance of 0.9912561385992329 to a new imbalance of 4.278313442226764E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6788888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 23:03:01,453 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1105347588:90 , srv1152293358:90 , srv1163855510:90 , srv1181170312:90 , srv1209382876:90 , srv1373901155:90 , srv1444630992:90 , srv1587468722:90 , srv1618131291:90 , srv1631318319:90 , srv1646718812:90 , srv1692718046:90 , srv2004045914:90 , srv2015476579:90 , srv2054721217:90 , srv2089304497:90 , srv24323774:90 , srv268246896:90 , srv336783785:90 , srv355265103:90 , srv409702318:90 , srv445318292:90 , srv53282439:90 , srv539436965:90 , srv614284600:90 , srv671815061:90 , srv731027329:90 , srv823221781:90 , srv937162167:90 , srv953623969:90 } 2023-03-31 23:03:01,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-31 23:03:01,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv731027329=26, srv614284600=24, srv1181170312=3, srv24323774=16, srv1618131291=8, srv1646718812=10, srv409702318=20, srv2015476579=13, srv355265103=19, srv1152293358=1, srv953623969=29, srv336783785=18, srv2054721217=14, srv1209382876=4, srv937162167=28, srv1444630992=6, srv1631318319=9, srv1373901155=5, srv2089304497=15, srv1692718046=11, srv539436965=23, srv1163855510=2, srv2004045914=12, srv445318292=21, srv1105347588=0, srv671815061=25, srv53282439=22, srv1587468722=7, srv823221781=27, srv268246896=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-31 23:03:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-31 23:03:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-31 23:03:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-31 23:03:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-31 23:03:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-31 23:03:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-31 23:03:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-31 23:03:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-31 23:03:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-31 23:03:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-31 23:03:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-31 23:03:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-31 23:03:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-31 23:03:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-31 23:03:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-31 23:03:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-31 23:03:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-31 23:03:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-31 23:03:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-31 23:03:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-31 23:03:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-31 23:03:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-31 23:03:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-31 23:03:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-31 23:03:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-31 23:03:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-31 23:03:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-31 23:03:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-31 23:03:01,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-31 23:03:01,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-31 23:03:01,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-03-31 23:03:01,469 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-31 23:03:01,469 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1105347588:90 , srv1152293358:90 , srv1163855510:90 , srv1181170312:90 , srv1209382876:90 , srv1373901155:90 , srv1444630992:90 , srv1587468722:90 , srv1618131291:90 , srv1631318319:90 , srv1646718812:90 , srv1692718046:90 , srv2004045914:90 , srv2015476579:90 , srv2054721217:90 , srv2089304497:90 , srv24323774:90 , srv268246896:90 , srv336783785:90 , srv355265103:90 , srv409702318:90 , srv445318292:90 , srv53282439:90 , srv539436965:90 , srv614284600:90 , srv671815061:90 , srv731027329:90 , srv823221781:90 , srv937162167:90 , srv953623969:90 } 2023-03-31 23:03:01,479 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=193 (was 280), ProcessCount=162 (was 164), AvailableMemoryMB=9811 (was 8898) - AvailableMemoryMB LEAK? -