2024-04-24 02:59:23,036 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-04-24 02:59:23,270 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-04-24 02:59:23,361 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-04-24 02:59:23,416 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-24 02:59:23,425 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-04-24 02:59:23,439 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=289, MaxFileDescriptor=60000, SystemLoadAverage=271, ProcessCount=175, AvailableMemoryMB=1532 2024-04-24 02:59:23,441 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-24 02:59:23,441 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-04-24 02:59:23,546 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1033537789:84 , srv1076290326:84 , srv108037831:84 , srv1085152030:84 , srv1139259142:84 , srv1161315437:84 , srv1212648768:84 , srv1214623330:84 , srv1225970200:84 , srv1249622077:84 , srv1276401563:84 , srv1287517341:84 , srv1320178940:84 , srv139545185:84 , srv1409515232:84 , srv1419344214:84 , srv1468825263:84 , srv1480115476:84 , srv1489310191:84 , srv1502791215:84 , srv1511701382:84 , srv1523299964:84 , srv1566066817:84 , srv1570901552:84 , srv1596969724:84 , srv1605271804:84 , srv1621688124:84 , srv1622267419:84 , srv1631142360:84 , srv1639193050:84 , srv1645050225:84 , srv1647250331:84 , srv1658147495:84 , srv1664870787:84 , srv1667339687:84 , srv1682119116:84 , srv1685501199:84 , srv1776757656:84 , srv1814046579:84 , srv1817014883:84 , srv1843504875:84 , srv1848441111:84 , srv1854973853:84 , srv1859297868:84 , srv1867621239:84 , srv1867890949:84 , srv1887562426:84 , srv1903755673:84 , srv1946566089:84 , srv1980304783:84 , srv1995186363:84 , srv2002235962:84 , srv2011980289:84 , srv2017762344:84 , srv2039709047:84 , srv2057459309:84 , srv2077016093:84 , srv2106614530:84 , srv2132627537:84 , srv2145725364:84 , srv220622173:84 , srv232951792:84 , srv253483859:84 , srv274349056:84 , srv277030715:84 , srv290908693:84 , srv296713762:84 , srv308978974:84 , srv330572079:84 , srv358714489:84 , srv408331310:84 , srv422152697:84 , srv436915387:84 , srv441355837:84 , srv454738555:84 , srv459798319:84 , srv482756273:84 , srv505073586:84 , srv507013282:84 , srv513071569:84 , srv529530908:84 , srv533403961:84 , srv552096231:84 , srv589740113:84 , srv645542636:84 , srv645891311:84 , srv666779061:84 , srv672592680:84 , srv735250583:84 , srv768064827:84 , srv782131121:84 , srv808180111:84 , srv82439974:84 , srv835167428:84 , srv850249667:84 , srv885954080:84 , srv887833102:84 , srv948066657:84 , srv971299329:84 , srv219353439:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-04-24 02:59:23,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-24 02:59:23,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1085152030=3, srv1814046579=38, srv1867890949=45, srv1685501199=36, srv290908693=66, srv422152697=72, srv1995186363=50, srv358714489=70, srv1480115476=17, srv645542636=85, srv1214623330=7, srv1502791215=19, srv835167428=94, srv1843504875=40, srv666779061=87, srv1468825263=16, srv1511701382=20, srv1980304783=49, srv1225970200=8, srv1566066817=22, srv2077016093=56, srv1419344214=15, srv768064827=90, srv645891311=86, srv1867621239=44, srv454738555=75, srv885954080=96, srv296713762=67, srv533403961=82, srv330572079=69, srv1645050225=30, srv1621688124=26, srv2106614530=57, srv459798319=76, srv108037831=2, srv2057459309=55, srv505073586=78, srv1667339687=34, srv1276401563=10, srv589740113=84, srv1647250331=31, srv808180111=92, srv672592680=88, srv277030715=65, srv1033537789=0, srv1776757656=37, srv219353439=60, srv482756273=77, srv1658147495=32, srv1489310191=18, srv2145725364=59, srv1076290326=1, srv1212648768=6, srv1523299964=21, srv513071569=80, srv735250583=89, srv1320178940=12, srv1664870787=33, srv1848441111=41, srv1596969724=24, srv1622267419=27, srv436915387=73, srv1139259142=4, srv971299329=99, srv1903755673=47, srv1161315437=5, srv2011980289=52, srv887833102=97, srv408331310=71, srv139545185=13, srv1817014883=39, srv529530908=81, srv1854973853=42, srv2132627537=58, srv1682119116=35, srv82439974=93, srv1249622077=9, srv441355837=74, srv1946566089=48, srv1631142360=28, srv253483859=63, srv2039709047=54, srv1287517341=11, srv850249667=95, srv948066657=98, srv2002235962=51, srv1605271804=25, srv232951792=62, srv1887562426=46, srv1570901552=23, srv274349056=64, srv220622173=61, srv2017762344=53, srv507013282=79, srv782131121=91, srv1409515232=14, srv1639193050=29, srv1859297868=43, srv552096231=83, srv308978974=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-24 02:59:23,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-24 02:59:23,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-24 02:59:23,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-24 02:59:23,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-24 02:59:23,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-24 02:59:23,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-24 02:59:23,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-24 02:59:23,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-24 02:59:23,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-24 02:59:23,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-24 02:59:23,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-24 02:59:23,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-24 02:59:23,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-24 02:59:23,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-24 02:59:23,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-24 02:59:23,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-24 02:59:23,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-24 02:59:23,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-24 02:59:23,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-24 02:59:23,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-24 02:59:23,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-24 02:59:23,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-24 02:59:23,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-24 02:59:23,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-24 02:59:23,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-24 02:59:23,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-24 02:59:23,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-24 02:59:23,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-24 02:59:23,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-24 02:59:23,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-24 02:59:23,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-24 02:59:23,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-24 02:59:23,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-24 02:59:23,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-24 02:59:23,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-24 02:59:23,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-24 02:59:23,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-24 02:59:23,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-24 02:59:23,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-24 02:59:23,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-24 02:59:23,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-24 02:59:23,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-24 02:59:23,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-24 02:59:23,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-24 02:59:23,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-24 02:59:23,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-24 02:59:23,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-24 02:59:23,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-24 02:59:23,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-24 02:59:23,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-24 02:59:23,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-24 02:59:23,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-24 02:59:23,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-24 02:59:23,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-24 02:59:23,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-24 02:59:23,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-24 02:59:23,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-24 02:59:23,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-24 02:59:23,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-24 02:59:23,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-24 02:59:23,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-24 02:59:23,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-24 02:59:23,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-24 02:59:23,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-24 02:59:23,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-24 02:59:23,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-24 02:59:23,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-24 02:59:23,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-24 02:59:23,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-24 02:59:23,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-24 02:59:23,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-24 02:59:23,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-24 02:59:23,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-24 02:59:23,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-24 02:59:23,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-24 02:59:23,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-24 02:59:23,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-24 02:59:23,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-24 02:59:23,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-24 02:59:23,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-24 02:59:23,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-24 02:59:23,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-24 02:59:23,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-24 02:59:23,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-24 02:59:23,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-24 02:59:23,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-24 02:59:23,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-24 02:59:23,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-24 02:59:23,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-24 02:59:23,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-24 02:59:23,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-24 02:59:23,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-24 02:59:23,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-24 02:59:23,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-24 02:59:23,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-24 02:59:23,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-24 02:59:23,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-24 02:59:23,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-24 02:59:23,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-24 02:59:23,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-24 02:59:23,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-24 02:59:23,766 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-24 02:59:23,766 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-24 02:59:23,766 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-24 02:59:24,166 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 508 ms to try 5000 different iterations. Found a solution that moves 4719 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17544631839255592. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041603356316109674, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05924515825496159, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5243333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041603356316109674, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17041666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23908333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-24 02:59:24,187 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1817014883:82 , srv666779061:82 , srv1249622077:83 , srv1419344214:83 , srv1867890949:83 , srv1995186363:83 , srv330572079:83 , srv505073586:83 , srv645891311:83 , srv672592680:83 , srv108037831:84 , srv1468825263:84 , srv1566066817:84 , srv1682119116:84 , srv1776757656:84 , srv1814046579:84 , srv1848441111:84 , srv2145725364:84 , srv253483859:84 , srv482756273:84 , srv850249667:84 , srv948066657:84 , srv1214623330:85 , srv1287517341:85 , srv1502791215:85 , srv1523299964:85 , srv1570901552:85 , srv1605271804:85 , srv1622267419:85 , srv1658147495:85 , srv1843504875:85 , srv1946566089:85 , srv1980304783:85 , srv2039709047:85 , srv2106614530:85 , srv2132627537:85 , srv358714489:85 , srv507013282:85 , srv529530908:85 , srv533403961:85 , srv552096231:85 , srv808180111:85 , srv1033537789:86 , srv1276401563:86 , srv139545185:86 , srv1511701382:86 , srv1631142360:86 , srv1647250331:86 , srv1667339687:86 , srv1859297868:86 , srv1867621239:86 , srv2077016093:86 , srv220622173:86 , srv232951792:86 , srv274349056:86 , srv277030715:86 , srv290908693:86 , srv645542636:86 , srv82439974:86 , srv887833102:86 , srv1076290326:87 , srv1161315437:87 , srv1225970200:87 , srv1480115476:87 , srv1621688124:87 , srv1645050225:87 , srv1887562426:87 , srv2057459309:87 , srv296713762:87 , srv513071569:87 , srv782131121:87 , srv835167428:87 , srv971299329:87 , srv1854973853:88 , srv2002235962:88 , srv589740113:88 , srv735250583:88 , srv768064827:88 , srv1139259142:89 , srv1212648768:89 , srv1596969724:89 , srv1639193050:89 , srv1664870787:89 , srv1685501199:89 , srv2017762344:89 , srv408331310:89 , srv422152697:89 , srv436915387:89 , srv1085152030:90 , srv1320178940:90 , srv1903755673:90 , srv308978974:90 , srv441355837:90 , srv454738555:90 , srv459798319:90 , srv885954080:90 , srv2011980289:91 , srv1409515232:92 , srv1489310191:92 , srv219353439:460 } 2024-04-24 02:59:24,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-24 02:59:24,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1085152030=3, srv1814046579=38, srv1867890949=45, srv1685501199=36, srv290908693=66, srv422152697=72, srv1995186363=50, srv358714489=70, srv1480115476=17, srv645542636=85, srv1214623330=7, srv1502791215=19, srv835167428=94, srv1843504875=40, srv666779061=87, srv1468825263=16, srv1511701382=20, srv1980304783=49, srv1225970200=8, srv1566066817=22, srv2077016093=56, srv1419344214=15, srv768064827=90, srv645891311=86, srv1867621239=44, srv454738555=75, srv885954080=96, srv296713762=67, srv533403961=82, srv330572079=69, srv1645050225=30, srv1621688124=26, srv2106614530=57, srv459798319=76, srv108037831=2, srv2057459309=55, srv505073586=78, srv1667339687=34, srv1276401563=10, srv589740113=84, srv1647250331=31, srv808180111=92, srv672592680=88, srv277030715=65, srv1033537789=0, srv1776757656=37, srv219353439=60, srv482756273=77, srv1658147495=32, srv1489310191=18, srv2145725364=59, srv1076290326=1, srv1212648768=6, srv1523299964=21, srv513071569=80, srv735250583=89, srv1320178940=12, srv1664870787=33, srv1848441111=41, srv1596969724=24, srv1622267419=27, srv436915387=73, srv1139259142=4, srv971299329=99, srv1903755673=47, srv1161315437=5, srv2011980289=52, srv887833102=97, srv408331310=71, srv139545185=13, srv1817014883=39, srv529530908=81, srv1854973853=42, srv2132627537=58, srv1682119116=35, srv82439974=93, srv1249622077=9, srv441355837=74, srv1946566089=48, srv1631142360=28, srv253483859=63, srv2039709047=54, srv1287517341=11, srv850249667=95, srv948066657=98, srv2002235962=51, srv1605271804=25, srv232951792=62, srv1887562426=46, srv1570901552=23, srv274349056=64, srv220622173=61, srv2017762344=53, srv507013282=79, srv782131121=91, srv1409515232=14, srv1639193050=29, srv1859297868=43, srv552096231=83, srv308978974=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-24 02:59:24,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-24 02:59:24,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-24 02:59:24,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-24 02:59:24,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-24 02:59:24,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-24 02:59:24,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-24 02:59:24,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-24 02:59:24,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-24 02:59:24,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-24 02:59:24,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-24 02:59:24,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-24 02:59:24,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-24 02:59:24,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-24 02:59:24,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-24 02:59:24,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-24 02:59:24,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-24 02:59:24,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-24 02:59:24,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-24 02:59:24,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-24 02:59:24,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-24 02:59:24,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-24 02:59:24,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-24 02:59:24,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-24 02:59:24,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-24 02:59:24,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-24 02:59:24,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-24 02:59:24,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-24 02:59:24,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-24 02:59:24,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-24 02:59:24,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-24 02:59:24,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-24 02:59:24,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-24 02:59:24,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-24 02:59:24,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-24 02:59:24,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-24 02:59:24,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-24 02:59:24,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-24 02:59:24,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-24 02:59:24,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-24 02:59:24,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-24 02:59:24,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-24 02:59:24,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-24 02:59:24,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-24 02:59:24,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-24 02:59:24,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-24 02:59:24,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-24 02:59:24,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-24 02:59:24,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-24 02:59:24,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-24 02:59:24,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-24 02:59:24,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-24 02:59:24,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-24 02:59:24,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-24 02:59:24,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-24 02:59:24,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-24 02:59:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-24 02:59:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-24 02:59:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-24 02:59:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-24 02:59:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-24 02:59:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-24 02:59:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-24 02:59:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-24 02:59:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-24 02:59:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-24 02:59:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-24 02:59:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-24 02:59:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-24 02:59:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-24 02:59:24,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-24 02:59:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-24 02:59:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-24 02:59:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-24 02:59:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-24 02:59:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-24 02:59:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-24 02:59:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-24 02:59:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-24 02:59:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-24 02:59:24,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-24 02:59:24,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-24 02:59:24,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-24 02:59:24,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-24 02:59:24,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-24 02:59:24,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-24 02:59:24,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-24 02:59:24,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-24 02:59:24,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-24 02:59:24,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-24 02:59:24,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-24 02:59:24,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-24 02:59:24,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-24 02:59:24,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-24 02:59:24,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-24 02:59:24,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-24 02:59:24,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-24 02:59:24,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-24 02:59:24,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-24 02:59:24,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-24 02:59:24,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-24 02:59:24,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-24 02:59:24,318 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041603356316109674, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05924515825496159, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041603356316109674, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17041666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23908333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-24 02:59:24,318 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-24 02:59:24,319 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1754132752483106, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041603356316109674, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05924515825496159, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041603356316109674, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17041666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23908333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-24 02:59:24,480 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 240 ms to try 5000 different iterations. Found a solution that moves 2630 regions; Going from a computed imbalance of 0.1754132752483106 to a new imbalance of 0.015031770295100956. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02377147298856112); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03484131092898602, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2922222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02377147298856112); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04708333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-24 02:59:24,489 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1409515232:78 , srv2011980289:80 , srv885954080:80 , srv1085152030:81 , srv1489310191:81 , srv1903755673:81 , srv408331310:81 , srv1639193050:82 , srv422152697:82 , srv436915387:82 , srv768064827:82 , srv971299329:82 , srv1212648768:83 , srv1276401563:83 , srv1320178940:83 , srv1480115476:83 , srv1511701382:83 , srv1854973853:83 , srv2077016093:83 , srv274349056:83 , srv308978974:83 , srv459798319:83 , srv1076290326:84 , srv1139259142:84 , srv1647250331:84 , srv1658147495:84 , srv1667339687:84 , srv1887562426:84 , srv220622173:84 , srv296713762:84 , srv454738555:84 , srv1033537789:85 , srv1225970200:85 , srv139545185:85 , srv1685501199:85 , srv1814046579:85 , srv1848441111:85 , srv1980304783:85 , srv2057459309:85 , srv2106614530:85 , srv290908693:85 , srv441355837:85 , srv533403961:85 , srv552096231:85 , srv589740113:85 , srv808180111:85 , srv1161315437:86 , srv1214623330:86 , srv1523299964:86 , srv1622267419:86 , srv1631142360:86 , srv1843504875:86 , srv1867890949:86 , srv2017762344:86 , srv2132627537:86 , srv513071569:86 , srv735250583:86 , srv782131121:86 , srv82439974:86 , srv887833102:86 , srv948066657:86 , srv1249622077:87 , srv1287517341:87 , srv1566066817:87 , srv1570901552:87 , srv1596969724:87 , srv1621688124:87 , srv1645050225:87 , srv1664870787:87 , srv1776757656:87 , srv2002235962:87 , srv277030715:87 , srv330572079:87 , srv482756273:87 , srv645542636:87 , srv850249667:87 , srv1468825263:88 , srv1605271804:88 , srv1682119116:88 , srv1859297868:88 , srv1867621239:88 , srv2039709047:88 , srv2145725364:88 , srv358714489:88 , srv505073586:88 , srv529530908:88 , srv666779061:88 , srv1419344214:89 , srv1502791215:89 , srv1946566089:89 , srv232951792:89 , srv672592680:89 , srv835167428:89 , srv108037831:90 , srv1995186363:90 , srv253483859:90 , srv507013282:91 , srv1817014883:92 , srv645891311:92 , srv219353439:525 } 2024-04-24 02:59:24,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-24 02:59:24,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1085152030=3, srv1814046579=38, srv1867890949=45, srv1685501199=36, srv290908693=66, srv422152697=72, srv1995186363=50, srv358714489=70, srv1480115476=17, srv645542636=85, srv1214623330=7, srv1502791215=19, srv835167428=94, srv1843504875=40, srv666779061=87, srv1468825263=16, srv1511701382=20, srv1980304783=49, srv1225970200=8, srv1566066817=22, srv2077016093=56, srv1419344214=15, srv768064827=90, srv645891311=86, srv1867621239=44, srv454738555=75, srv885954080=96, srv296713762=67, srv533403961=82, srv330572079=69, srv1645050225=30, srv1621688124=26, srv2106614530=57, srv459798319=76, srv108037831=2, srv2057459309=55, srv505073586=78, srv1667339687=34, srv1276401563=10, srv589740113=84, srv1647250331=31, srv808180111=92, srv672592680=88, srv277030715=65, srv1033537789=0, srv1776757656=37, srv219353439=60, srv482756273=77, srv1658147495=32, srv1489310191=18, srv2145725364=59, srv1076290326=1, srv1212648768=6, srv1523299964=21, srv513071569=80, srv735250583=89, srv1320178940=12, srv1664870787=33, srv1848441111=41, srv1596969724=24, srv1622267419=27, srv436915387=73, srv1139259142=4, srv971299329=99, srv1903755673=47, srv1161315437=5, srv2011980289=52, srv887833102=97, srv408331310=71, srv139545185=13, srv1817014883=39, srv529530908=81, srv1854973853=42, srv2132627537=58, srv1682119116=35, srv82439974=93, srv1249622077=9, srv441355837=74, srv1946566089=48, srv1631142360=28, srv253483859=63, srv2039709047=54, srv1287517341=11, srv850249667=95, srv948066657=98, srv2002235962=51, srv1605271804=25, srv232951792=62, srv1887562426=46, srv1570901552=23, srv274349056=64, srv220622173=61, srv2017762344=53, srv507013282=79, srv782131121=91, srv1409515232=14, srv1639193050=29, srv1859297868=43, srv552096231=83, srv308978974=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-24 02:59:24,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-24 02:59:24,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-24 02:59:24,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-24 02:59:24,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-24 02:59:24,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-24 02:59:24,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-24 02:59:24,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-24 02:59:24,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-24 02:59:24,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-24 02:59:24,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-24 02:59:24,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-24 02:59:24,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-24 02:59:24,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-24 02:59:24,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-24 02:59:24,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-24 02:59:24,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-24 02:59:24,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-24 02:59:24,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-24 02:59:24,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-24 02:59:24,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-24 02:59:24,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-24 02:59:24,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-24 02:59:24,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-24 02:59:24,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-24 02:59:24,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-24 02:59:24,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-24 02:59:24,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-24 02:59:24,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-24 02:59:24,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-24 02:59:24,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-24 02:59:24,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-24 02:59:24,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-24 02:59:24,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-24 02:59:24,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-24 02:59:24,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-24 02:59:24,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-24 02:59:24,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-24 02:59:24,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-24 02:59:24,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-24 02:59:24,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-24 02:59:24,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-24 02:59:24,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-24 02:59:24,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-24 02:59:24,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-24 02:59:24,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-24 02:59:24,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-24 02:59:24,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-24 02:59:24,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-24 02:59:24,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-24 02:59:24,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-24 02:59:24,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-24 02:59:24,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-24 02:59:24,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-24 02:59:24,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-24 02:59:24,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-24 02:59:24,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-24 02:59:24,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-24 02:59:24,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-24 02:59:24,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-24 02:59:24,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-24 02:59:24,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-24 02:59:24,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-24 02:59:24,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-24 02:59:24,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-24 02:59:24,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-24 02:59:24,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-24 02:59:24,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-24 02:59:24,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-24 02:59:24,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-24 02:59:24,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-24 02:59:24,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-24 02:59:24,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-24 02:59:24,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-24 02:59:24,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-24 02:59:24,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-24 02:59:24,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-24 02:59:24,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-24 02:59:24,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-24 02:59:24,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-24 02:59:24,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-24 02:59:24,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-24 02:59:24,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-24 02:59:24,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-24 02:59:24,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-24 02:59:24,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-24 02:59:24,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-24 02:59:24,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-24 02:59:24,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-24 02:59:24,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-24 02:59:24,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-24 02:59:24,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-24 02:59:24,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-24 02:59:24,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-24 02:59:24,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-24 02:59:24,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-24 02:59:24,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-24 02:59:24,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-24 02:59:24,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-24 02:59:24,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-24 02:59:24,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-24 02:59:24,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-24 02:59:24,618 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02377147298856112); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03484131092898602, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02377147298856112); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04708333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-24 02:59:24,618 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-24 02:59:24,618 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015013354641495297, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02377147298856112); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03484131092898602, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02377147298856112); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04708333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-24 02:59:24,768 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 235 ms to try 5000 different iterations. Found a solution that moves 1019 regions; Going from a computed imbalance of 0.015013354641495297 to a new imbalance of 4.2967687732462114E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007332959212304937); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009356392870229827); MoveCostFunction : (multiplier=7.0, imbalance=0.11322222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007332959212304937); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-24 02:59:24,769 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1596969724:81 , srv2017762344:81 , srv232951792:81 , srv835167428:81 , srv1664870787:82 , srv1867621239:82 , srv2002235962:82 , srv507013282:82 , srv645891311:82 , srv1502791215:83 , srv1685501199:83 , srv1859297868:83 , srv1946566089:83 , srv441355837:83 , srv454738555:83 , srv513071569:83 , srv735250583:83 , srv1139259142:84 , srv1161315437:84 , srv1489310191:84 , srv1570901552:84 , srv1621688124:84 , srv1645050225:84 , srv1817014883:84 , srv1995186363:84 , srv2039709047:84 , srv253483859:84 , srv277030715:84 , srv308978974:84 , srv358714489:84 , srv645542636:84 , srv782131121:84 , srv108037831:85 , srv1225970200:85 , srv1287517341:85 , srv1320178940:85 , srv1468825263:85 , srv1605271804:85 , srv1682119116:85 , srv2057459309:85 , srv2145725364:85 , srv459798319:85 , srv529530908:85 , srv589740113:85 , srv672592680:85 , srv82439974:85 , srv1085152030:86 , srv1212648768:86 , srv1214623330:86 , srv139545185:86 , srv1631142360:86 , srv1639193050:86 , srv1667339687:86 , srv2011980289:86 , srv2132627537:86 , srv290908693:86 , srv296713762:86 , srv408331310:86 , srv422152697:86 , srv436915387:86 , srv887833102:86 , srv1033537789:87 , srv1076290326:87 , srv1249622077:87 , srv1419344214:87 , srv1480115476:87 , srv1523299964:87 , srv1566066817:87 , srv1622267419:87 , srv1776757656:87 , srv1843504875:87 , srv1854973853:87 , srv1887562426:87 , srv1903755673:87 , srv2106614530:87 , srv220622173:87 , srv274349056:87 , srv330572079:87 , srv482756273:87 , srv505073586:87 , srv666779061:87 , srv768064827:87 , srv808180111:87 , srv885954080:87 , srv1409515232:88 , srv1511701382:88 , srv1647250331:88 , srv1658147495:88 , srv1814046579:88 , srv1848441111:88 , srv1980304783:88 , srv533403961:88 , srv552096231:88 , srv850249667:88 , srv948066657:88 , srv971299329:88 , srv1276401563:89 , srv1867890949:89 , srv2077016093:89 , srv219353439:538 } 2024-04-24 02:59:24,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-24 02:59:24,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1085152030=3, srv1814046579=38, srv1867890949=45, srv1685501199=36, srv290908693=66, srv422152697=72, srv1995186363=50, srv358714489=70, srv1480115476=17, srv645542636=85, srv1214623330=7, srv1502791215=19, srv835167428=94, srv1843504875=40, srv666779061=87, srv1468825263=16, srv1511701382=20, srv1980304783=49, srv1225970200=8, srv1566066817=22, srv2077016093=56, srv1419344214=15, srv768064827=90, srv645891311=86, srv1867621239=44, srv454738555=75, srv885954080=96, srv296713762=67, srv533403961=82, srv330572079=69, srv1645050225=30, srv1621688124=26, srv2106614530=57, srv459798319=76, srv108037831=2, srv2057459309=55, srv505073586=78, srv1667339687=34, srv1276401563=10, srv589740113=84, srv1647250331=31, srv808180111=92, srv672592680=88, srv277030715=65, srv1033537789=0, srv1776757656=37, srv219353439=60, srv482756273=77, srv1658147495=32, srv1489310191=18, srv2145725364=59, srv1076290326=1, srv1212648768=6, srv1523299964=21, srv513071569=80, srv735250583=89, srv1320178940=12, srv1664870787=33, srv1848441111=41, srv1596969724=24, srv1622267419=27, srv436915387=73, srv1139259142=4, srv971299329=99, srv1903755673=47, srv1161315437=5, srv2011980289=52, srv887833102=97, srv408331310=71, srv139545185=13, srv1817014883=39, srv529530908=81, srv1854973853=42, srv2132627537=58, srv1682119116=35, srv82439974=93, srv1249622077=9, srv441355837=74, srv1946566089=48, srv1631142360=28, srv253483859=63, srv2039709047=54, srv1287517341=11, srv850249667=95, srv948066657=98, srv2002235962=51, srv1605271804=25, srv232951792=62, srv1887562426=46, srv1570901552=23, srv274349056=64, srv220622173=61, srv2017762344=53, srv507013282=79, srv782131121=91, srv1409515232=14, srv1639193050=29, srv1859297868=43, srv552096231=83, srv308978974=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-24 02:59:24,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-24 02:59:24,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-24 02:59:24,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-24 02:59:24,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-24 02:59:24,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-24 02:59:24,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-24 02:59:24,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-24 02:59:24,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-24 02:59:24,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-24 02:59:24,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-24 02:59:24,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-24 02:59:24,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-24 02:59:24,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-24 02:59:24,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-24 02:59:24,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-24 02:59:24,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-24 02:59:24,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-24 02:59:24,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-24 02:59:24,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-24 02:59:24,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-24 02:59:24,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-24 02:59:24,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-24 02:59:24,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-24 02:59:24,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-24 02:59:24,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-24 02:59:24,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-24 02:59:24,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-24 02:59:24,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-24 02:59:24,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-24 02:59:24,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-24 02:59:24,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-24 02:59:24,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-24 02:59:24,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-24 02:59:24,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-24 02:59:24,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-24 02:59:24,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-24 02:59:24,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-24 02:59:24,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-24 02:59:24,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-24 02:59:24,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-24 02:59:24,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-24 02:59:24,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-24 02:59:24,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-24 02:59:24,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-24 02:59:24,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-24 02:59:24,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-24 02:59:24,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-24 02:59:24,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-24 02:59:24,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-24 02:59:24,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-24 02:59:24,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-24 02:59:24,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-24 02:59:24,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-24 02:59:24,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-24 02:59:24,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-24 02:59:24,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-24 02:59:24,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-24 02:59:24,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-24 02:59:24,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-24 02:59:24,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-24 02:59:24,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-24 02:59:24,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-24 02:59:24,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-24 02:59:24,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-24 02:59:24,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-24 02:59:24,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-24 02:59:24,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-24 02:59:24,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-24 02:59:24,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-24 02:59:24,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-24 02:59:24,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-24 02:59:24,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-24 02:59:24,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-24 02:59:24,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-24 02:59:24,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-24 02:59:24,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-24 02:59:24,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-24 02:59:24,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-24 02:59:24,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-24 02:59:24,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-24 02:59:24,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-24 02:59:24,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-24 02:59:24,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-24 02:59:24,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-24 02:59:24,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-24 02:59:24,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-24 02:59:24,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-24 02:59:24,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-24 02:59:24,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-24 02:59:24,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-24 02:59:24,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-24 02:59:24,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-24 02:59:24,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-24 02:59:24,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-24 02:59:24,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-24 02:59:24,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-24 02:59:24,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-24 02:59:24,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-24 02:59:24,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-24 02:59:24,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-24 02:59:24,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-24 02:59:24,869 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007332959212304937); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009356392870229827); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007332959212304937); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-24 02:59:24,869 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-24 02:59:24,870 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.225416868211366E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007332959212304937); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009356392870229827); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007332959212304937); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-24 02:59:25,098 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 296 ms to try 5000 different iterations. Found a solution that moves 223 regions; Going from a computed imbalance of 4.225416868211366E-4 to a new imbalance of 1.8744336480962283E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.024777777777777777); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-24 02:59:25,099 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv108037831:83 , srv1419344214:83 , srv850249667:83 , srv1033537789:84 , srv1076290326:84 , srv1212648768:84 , srv1225970200:84 , srv1276401563:84 , srv1287517341:84 , srv1320178940:84 , srv1409515232:84 , srv1468825263:84 , srv1523299964:84 , srv1566066817:84 , srv1605271804:84 , srv1621688124:84 , srv1622267419:84 , srv1645050225:84 , srv1647250331:84 , srv1664870787:84 , srv1776757656:84 , srv1817014883:84 , srv1843504875:84 , srv1854973853:84 , srv1867890949:84 , srv1887562426:84 , srv1903755673:84 , srv1946566089:84 , srv1980304783:84 , srv2077016093:84 , srv253483859:84 , srv422152697:84 , srv441355837:84 , srv459798319:84 , srv482756273:84 , srv507013282:84 , srv529530908:84 , srv533403961:84 , srv552096231:84 , srv589740113:84 , srv887833102:84 , srv948066657:84 , srv1085152030:85 , srv1139259142:85 , srv1161315437:85 , srv1214623330:85 , srv1249622077:85 , srv139545185:85 , srv1480115476:85 , srv1489310191:85 , srv1502791215:85 , srv1511701382:85 , srv1596969724:85 , srv1631142360:85 , srv1658147495:85 , srv1685501199:85 , srv1814046579:85 , srv1848441111:85 , srv1859297868:85 , srv1995186363:85 , srv2002235962:85 , srv2011980289:85 , srv2039709047:85 , srv2057459309:85 , srv2106614530:85 , srv2132627537:85 , srv2145725364:85 , srv220622173:85 , srv277030715:85 , srv290908693:85 , srv296713762:85 , srv308978974:85 , srv330572079:85 , srv358714489:85 , srv436915387:85 , srv454738555:85 , srv505073586:85 , srv645542636:85 , srv645891311:85 , srv666779061:85 , srv672592680:85 , srv735250583:85 , srv768064827:85 , srv782131121:85 , srv808180111:85 , srv82439974:85 , srv835167428:85 , srv885954080:85 , srv971299329:85 , srv1570901552:86 , srv1639193050:86 , srv1667339687:86 , srv1682119116:86 , srv1867621239:86 , srv2017762344:86 , srv232951792:86 , srv274349056:86 , srv408331310:86 , srv513071569:86 , srv219353439:620 } 2024-04-24 02:59:25,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-24 02:59:25,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1085152030=3, srv1814046579=38, srv1867890949=45, srv1685501199=36, srv290908693=66, srv422152697=72, srv1995186363=50, srv358714489=70, srv1480115476=17, srv645542636=85, srv1214623330=7, srv1502791215=19, srv835167428=94, srv1843504875=40, srv666779061=87, srv1468825263=16, srv1511701382=20, srv1980304783=49, srv1225970200=8, srv1566066817=22, srv2077016093=56, srv1419344214=15, srv768064827=90, srv645891311=86, srv1867621239=44, srv454738555=75, srv885954080=96, srv296713762=67, srv533403961=82, srv330572079=69, srv1645050225=30, srv1621688124=26, srv2106614530=57, srv459798319=76, srv108037831=2, srv2057459309=55, srv505073586=78, srv1667339687=34, srv1276401563=10, srv589740113=84, srv1647250331=31, srv808180111=92, srv672592680=88, srv277030715=65, srv1033537789=0, srv1776757656=37, srv219353439=60, srv482756273=77, srv1658147495=32, srv1489310191=18, srv2145725364=59, srv1076290326=1, srv1212648768=6, srv1523299964=21, srv513071569=80, srv735250583=89, srv1320178940=12, srv1664870787=33, srv1848441111=41, srv1596969724=24, srv1622267419=27, srv436915387=73, srv1139259142=4, srv971299329=99, srv1903755673=47, srv1161315437=5, srv2011980289=52, srv887833102=97, srv408331310=71, srv139545185=13, srv1817014883=39, srv529530908=81, srv1854973853=42, srv2132627537=58, srv1682119116=35, srv82439974=93, srv1249622077=9, srv441355837=74, srv1946566089=48, srv1631142360=28, srv253483859=63, srv2039709047=54, srv1287517341=11, srv850249667=95, srv948066657=98, srv2002235962=51, srv1605271804=25, srv232951792=62, srv1887562426=46, srv1570901552=23, srv274349056=64, srv220622173=61, srv2017762344=53, srv507013282=79, srv782131121=91, srv1409515232=14, srv1639193050=29, srv1859297868=43, srv552096231=83, srv308978974=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-24 02:59:25,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-24 02:59:25,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-24 02:59:25,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-24 02:59:25,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-24 02:59:25,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-24 02:59:25,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-24 02:59:25,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-24 02:59:25,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-24 02:59:25,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-24 02:59:25,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-24 02:59:25,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-24 02:59:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-24 02:59:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-24 02:59:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-24 02:59:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-24 02:59:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-24 02:59:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-24 02:59:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-24 02:59:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-24 02:59:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-24 02:59:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-24 02:59:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-24 02:59:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-24 02:59:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-24 02:59:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-24 02:59:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-24 02:59:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-24 02:59:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-24 02:59:25,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-24 02:59:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-24 02:59:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-24 02:59:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-24 02:59:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-24 02:59:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-24 02:59:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-24 02:59:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-24 02:59:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-24 02:59:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-24 02:59:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-24 02:59:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-24 02:59:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-24 02:59:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-24 02:59:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-24 02:59:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-24 02:59:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-24 02:59:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-24 02:59:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-24 02:59:25,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-24 02:59:25,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-24 02:59:25,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-24 02:59:25,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-24 02:59:25,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-24 02:59:25,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-24 02:59:25,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-24 02:59:25,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-24 02:59:25,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-24 02:59:25,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-24 02:59:25,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-24 02:59:25,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-24 02:59:25,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-24 02:59:25,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-24 02:59:25,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-24 02:59:25,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-24 02:59:25,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-24 02:59:25,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-24 02:59:25,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-24 02:59:25,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-24 02:59:25,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-24 02:59:25,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-24 02:59:25,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-24 02:59:25,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-24 02:59:25,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-24 02:59:25,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-24 02:59:25,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-24 02:59:25,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-24 02:59:25,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-24 02:59:25,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-24 02:59:25,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-24 02:59:25,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-24 02:59:25,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-24 02:59:25,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-24 02:59:25,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-24 02:59:25,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-24 02:59:25,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-24 02:59:25,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-24 02:59:25,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-24 02:59:25,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-24 02:59:25,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-24 02:59:25,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-24 02:59:25,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-24 02:59:25,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-24 02:59:25,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-24 02:59:25,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-24 02:59:25,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-24 02:59:25,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-24 02:59:25,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-24 02:59:25,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-24 02:59:25,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-24 02:59:25,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-24 02:59:25,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-24 02:59:25,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-24 02:59:25,204 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.7182857106794416E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-24 02:59:25,205 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv108037831:83 , srv1419344214:83 , srv850249667:83 , srv1033537789:84 , srv1076290326:84 , srv1212648768:84 , srv1225970200:84 , srv1276401563:84 , srv1287517341:84 , srv1320178940:84 , srv1409515232:84 , srv1468825263:84 , srv1523299964:84 , srv1566066817:84 , srv1605271804:84 , srv1621688124:84 , srv1622267419:84 , srv1645050225:84 , srv1647250331:84 , srv1664870787:84 , srv1776757656:84 , srv1817014883:84 , srv1843504875:84 , srv1854973853:84 , srv1867890949:84 , srv1887562426:84 , srv1903755673:84 , srv1946566089:84 , srv1980304783:84 , srv2077016093:84 , srv253483859:84 , srv422152697:84 , srv441355837:84 , srv459798319:84 , srv482756273:84 , srv507013282:84 , srv529530908:84 , srv533403961:84 , srv552096231:84 , srv589740113:84 , srv887833102:84 , srv948066657:84 , srv1085152030:85 , srv1139259142:85 , srv1161315437:85 , srv1214623330:85 , srv1249622077:85 , srv139545185:85 , srv1480115476:85 , srv1489310191:85 , srv1502791215:85 , srv1511701382:85 , srv1596969724:85 , srv1631142360:85 , srv1658147495:85 , srv1685501199:85 , srv1814046579:85 , srv1848441111:85 , srv1859297868:85 , srv1995186363:85 , srv2002235962:85 , srv2011980289:85 , srv2039709047:85 , srv2057459309:85 , srv2106614530:85 , srv2132627537:85 , srv2145725364:85 , srv220622173:85 , srv277030715:85 , srv290908693:85 , srv296713762:85 , srv308978974:85 , srv330572079:85 , srv358714489:85 , srv436915387:85 , srv454738555:85 , srv505073586:85 , srv645542636:85 , srv645891311:85 , srv666779061:85 , srv672592680:85 , srv735250583:85 , srv768064827:85 , srv782131121:85 , srv808180111:85 , srv82439974:85 , srv835167428:85 , srv885954080:85 , srv971299329:85 , srv1570901552:86 , srv1639193050:86 , srv1667339687:86 , srv1682119116:86 , srv1867621239:86 , srv2017762344:86 , srv232951792:86 , srv274349056:86 , srv408331310:86 , srv513071569:86 , srv219353439:620 } 2024-04-24 02:59:25,226 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=289 (was 289), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=271 (was 271), ProcessCount=175 (was 175), AvailableMemoryMB=909 (was 1532) 2024-04-24 02:59:25,234 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=289, MaxFileDescriptor=60000, SystemLoadAverage=271, ProcessCount=175, AvailableMemoryMB=909 2024-04-24 02:59:25,235 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-24 02:59:25,235 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-04-24 02:59:25,246 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1005423218:84 , srv1042840127:84 , srv1050122082:84 , srv1320559104:84 , srv1326817168:84 , srv1327457924:84 , srv1612173219:84 , srv1625931457:84 , srv164562473:84 , srv1805337648:84 , srv1880739238:84 , srv1943497481:84 , srv2026977284:84 , srv2078570279:84 , srv2102094595:84 , srv2113645601:84 , srv34865940:84 , srv355763382:84 , srv364115992:84 , srv448801982:84 , srv451884474:84 , srv486149544:84 , srv534210014:84 , srv570178651:84 , srv585401166:84 , srv76791671:84 , srv926140889:84 , srv955074034:84 , srv969217369:84 , srv874613092:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-04-24 02:59:25,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-24 02:59:25,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv570178651=23, srv451884474=20, srv534210014=22, srv926140889=27, srv2078570279=13, srv1327457924=5, srv1612173219=6, srv164562473=8, srv1326817168=4, srv34865940=16, srv1050122082=2, srv1625931457=7, srv2026977284=12, srv486149544=21, srv1805337648=9, srv2113645601=15, srv969217369=29, srv355763382=17, srv1005423218=0, srv874613092=26, srv1880739238=10, srv1042840127=1, srv955074034=28, srv448801982=19, srv76791671=25, srv1943497481=11, srv2102094595=14, srv364115992=18, srv585401166=24, srv1320559104=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-24 02:59:25,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-24 02:59:25,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-24 02:59:25,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-24 02:59:25,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-24 02:59:25,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-24 02:59:25,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-24 02:59:25,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-24 02:59:25,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-24 02:59:25,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-24 02:59:25,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-24 02:59:25,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-24 02:59:25,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-24 02:59:25,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-24 02:59:25,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-24 02:59:25,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-24 02:59:25,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-24 02:59:25,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-24 02:59:25,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-24 02:59:25,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-24 02:59:25,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-24 02:59:25,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-24 02:59:25,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-24 02:59:25,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-24 02:59:25,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-24 02:59:25,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-24 02:59:25,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-24 02:59:25,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-24 02:59:25,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-24 02:59:25,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-24 02:59:25,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-24 02:59:25,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-04-24 02:59:25,271 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1496571714985389, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-24 02:59:25,271 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912664608124915, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1496571714985389, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-04-24 03:01:25,261 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9701814 different iterations. Found a solution that moves 1848 regions; Going from a computed imbalance of 0.9912664608124915 to a new imbalance of 4.313324190526491E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6844444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-24 03:01:25,268 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1005423218:90 , srv1042840127:90 , srv1050122082:90 , srv1320559104:90 , srv1326817168:90 , srv1327457924:90 , srv1612173219:90 , srv1625931457:90 , srv164562473:90 , srv1805337648:90 , srv1880739238:90 , srv1943497481:90 , srv2026977284:90 , srv2078570279:90 , srv2102094595:90 , srv2113645601:90 , srv34865940:90 , srv355763382:90 , srv364115992:90 , srv448801982:90 , srv451884474:90 , srv486149544:90 , srv534210014:90 , srv570178651:90 , srv585401166:90 , srv76791671:90 , srv874613092:90 , srv926140889:90 , srv955074034:90 , srv969217369:90 } 2024-04-24 03:01:25,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-24 03:01:25,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv570178651=23, srv451884474=20, srv534210014=22, srv926140889=27, srv2078570279=13, srv1327457924=5, srv1612173219=6, srv164562473=8, srv1326817168=4, srv34865940=16, srv1050122082=2, srv1625931457=7, srv2026977284=12, srv486149544=21, srv1805337648=9, srv2113645601=15, srv969217369=29, srv355763382=17, srv1005423218=0, srv874613092=26, srv1880739238=10, srv1042840127=1, srv955074034=28, srv448801982=19, srv76791671=25, srv1943497481=11, srv2102094595=14, srv364115992=18, srv585401166=24, srv1320559104=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-24 03:01:25,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-24 03:01:25,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-24 03:01:25,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-24 03:01:25,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-24 03:01:25,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-24 03:01:25,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-24 03:01:25,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-24 03:01:25,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-24 03:01:25,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-24 03:01:25,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-24 03:01:25,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-24 03:01:25,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-24 03:01:25,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-24 03:01:25,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-24 03:01:25,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-24 03:01:25,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-24 03:01:25,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-24 03:01:25,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-24 03:01:25,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-24 03:01:25,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-24 03:01:25,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-24 03:01:25,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-24 03:01:25,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-24 03:01:25,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-24 03:01:25,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-24 03:01:25,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-24 03:01:25,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-24 03:01:25,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-24 03:01:25,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-24 03:01:25,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-24 03:01:25,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-04-24 03:01:25,286 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-24 03:01:25,286 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1005423218:90 , srv1042840127:90 , srv1050122082:90 , srv1320559104:90 , srv1326817168:90 , srv1327457924:90 , srv1612173219:90 , srv1625931457:90 , srv164562473:90 , srv1805337648:90 , srv1880739238:90 , srv1943497481:90 , srv2026977284:90 , srv2078570279:90 , srv2102094595:90 , srv2113645601:90 , srv34865940:90 , srv355763382:90 , srv364115992:90 , srv448801982:90 , srv451884474:90 , srv486149544:90 , srv534210014:90 , srv570178651:90 , srv585401166:90 , srv76791671:90 , srv874613092:90 , srv926140889:90 , srv955074034:90 , srv969217369:90 } 2024-04-24 03:01:25,299 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=289 (was 289), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=167 (was 271), ProcessCount=175 (was 175), AvailableMemoryMB=1666 (was 909) - AvailableMemoryMB LEAK? -