2024-02-11 19:53:09,296 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-02-11 19:53:09,598 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-02-11 19:53:09,717 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-02-11 19:53:09,780 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-11 19:53:09,787 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-02-11 19:53:09,800 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=270, ProcessCount=170, AvailableMemoryMB=3160 2024-02-11 19:53:09,802 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-11 19:53:09,802 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-02-11 19:53:09,935 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1032013644:84 , srv1034851246:84 , srv1073947484:84 , srv1171885443:84 , srv1213718417:84 , srv1266999196:84 , srv1270681839:84 , srv1282093968:84 , srv130382898:84 , srv1310021704:84 , srv1325455818:84 , srv1331128257:84 , srv134614334:84 , srv1348931118:84 , srv1370242462:84 , srv1370988946:84 , srv1374678319:84 , srv1434735727:84 , srv1444215503:84 , srv1471981274:84 , srv1494933506:84 , srv1496492581:84 , srv1505620968:84 , srv1508558671:84 , srv1523029196:84 , srv1539195426:84 , srv1553313638:84 , srv1565179182:84 , srv1572670677:84 , srv1616753329:84 , srv1630118433:84 , srv1645838413:84 , srv1657083072:84 , srv166004532:84 , srv1672845398:84 , srv1693061383:84 , srv1702806474:84 , srv1718080046:84 , srv172056254:84 , srv173365499:84 , srv1754055395:84 , srv1754549133:84 , srv1773531095:84 , srv1826603363:84 , srv1853755032:84 , srv1906222068:84 , srv1918015934:84 , srv1926023900:84 , srv1948357614:84 , srv1983189741:84 , srv1985252927:84 , srv1992427617:84 , srv1995087550:84 , srv2028925275:84 , srv205103833:84 , srv2060038489:84 , srv2091172568:84 , srv2104275896:84 , srv2118384420:84 , srv2137934854:84 , srv2144038819:84 , srv250770810:84 , srv269168731:84 , srv312237669:84 , srv378172006:84 , srv383836623:84 , srv384131706:84 , srv396598050:84 , srv408666192:84 , srv422134787:84 , srv466490222:84 , srv521089015:84 , srv536176949:84 , srv564048444:84 , srv579014209:84 , srv58208467:84 , srv602208863:84 , srv641145052:84 , srv67333051:84 , srv676991114:84 , srv694183813:84 , srv746183153:84 , srv755161500:84 , srv757452587:84 , srv788962913:84 , srv804694278:84 , srv81516030:84 , srv860239081:84 , srv867467580:84 , srv876735459:84 , srv884462157:84 , srv906809155:84 , srv922809520:84 , srv945323714:84 , srv94889449:84 , srv954127039:84 , srv962243138:84 , srv995851835:84 , srv996051775:84 , srv197327035:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-02-11 19:53:09,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 19:53:09,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1906222068=45, srv1754055395=40, srv1645838413=31, srv579014209=75, srv2028925275=54, srv1266999196=5, srv746183153=82, srv1325455818=10, srv94889449=95, srv564048444=74, srv1539195426=25, srv962243138=97, srv1918015934=46, srv1616753329=29, srv1630118433=30, srv1992427617=52, srv2060038489=56, srv1282093968=7, srv1471981274=19, srv1523029196=24, srv860239081=88, srv396598050=68, srv876735459=90, srv1553313638=26, srv166004532=33, srv173365499=39, srv1496492581=21, srv536176949=73, srv134614334=12, srv1693061383=35, srv788962913=85, srv1370988946=15, srv1983189741=50, srv2137934854=60, srv172056254=38, srv694183813=81, srv922809520=93, srv945323714=94, srv384131706=67, srv312237669=64, srv130382898=8, srv1702806474=36, srv1948357614=48, srv1853755032=44, srv58208467=76, srv1494933506=20, srv1718080046=37, srv1374678319=16, srv1348931118=13, srv521089015=72, srv1926023900=47, srv1657083072=32, srv2104275896=58, srv2118384420=59, srv676991114=80, srv197327035=49, srv1434735727=17, srv2091172568=57, srv2144038819=61, srv884462157=91, srv906809155=92, srv1331128257=11, srv1672845398=34, srv804694278=86, srv1565179182=27, srv1213718417=4, srv1505620968=22, srv205103833=55, srv422134787=70, srv995851835=98, srv1034851246=1, srv755161500=83, srv1572670677=28, srv1270681839=6, srv1826603363=43, srv269168731=63, srv602208863=77, srv954127039=96, srv1171885443=3, srv641145052=78, srv1754549133=41, srv378172006=65, srv757452587=84, srv1985252927=51, srv867467580=89, srv1444215503=18, srv408666192=69, srv67333051=79, srv466490222=71, srv996051775=99, srv1032013644=0, srv1370242462=14, srv1073947484=2, srv250770810=62, srv81516030=87, srv1773531095=42, srv383836623=66, srv1995087550=53, srv1310021704=9, srv1508558671=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 19:53:10,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 19:53:10,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 19:53:10,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 19:53:10,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 19:53:10,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 19:53:10,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 19:53:10,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 19:53:10,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 19:53:10,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 19:53:10,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 19:53:10,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 19:53:10,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 19:53:10,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 19:53:10,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 19:53:10,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 19:53:10,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 19:53:10,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 19:53:10,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 19:53:10,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 19:53:10,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 19:53:10,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 19:53:10,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 19:53:10,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 19:53:10,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 19:53:10,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 19:53:10,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 19:53:10,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 19:53:10,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 19:53:10,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 19:53:10,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 19:53:10,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 19:53:10,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 19:53:10,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 19:53:10,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 19:53:10,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 19:53:10,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 19:53:10,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 19:53:10,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 19:53:10,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 19:53:10,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 19:53:10,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 19:53:10,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 19:53:10,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 19:53:10,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 19:53:10,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 19:53:10,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 19:53:10,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 19:53:10,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 19:53:10,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 19:53:10,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 19:53:10,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 19:53:10,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 19:53:10,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 19:53:10,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 19:53:10,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 19:53:10,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 19:53:10,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 19:53:10,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 19:53:10,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 19:53:10,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 19:53:10,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 19:53:10,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 19:53:10,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 19:53:10,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 19:53:10,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 19:53:10,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 19:53:10,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 19:53:10,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 19:53:10,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 19:53:10,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 19:53:10,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 19:53:10,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 19:53:10,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 19:53:10,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 19:53:10,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 19:53:10,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 19:53:10,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 19:53:10,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 19:53:10,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 19:53:10,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 19:53:10,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 19:53:10,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 19:53:10,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 19:53:10,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 19:53:10,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 19:53:10,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 19:53:10,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 19:53:10,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 19:53:10,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 19:53:10,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 19:53:10,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 19:53:10,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 19:53:10,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 19:53:10,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 19:53:10,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 19:53:10,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 19:53:10,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 19:53:10,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 19:53:10,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 19:53:10,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 19:53:10,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 19:53:10,191 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 19:53:10,192 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-11 19:53:10,192 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-11 19:53:10,624 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 562 ms to try 5000 different iterations. Found a solution that moves 4718 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17528341683357637. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039834424737766716, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0549502376658065, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5242222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039834424737766716, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17033333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23841666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 19:53:10,645 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv384131706:81 , srv1853755032:82 , srv804694278:82 , srv1032013644:83 , srv1270681839:83 , srv1325455818:83 , srv134614334:83 , srv1370988946:83 , srv1496492581:83 , srv1572670677:83 , srv1693061383:83 , srv1826603363:83 , srv1918015934:83 , srv1985252927:83 , srv422134787:83 , srv996051775:83 , srv1374678319:84 , srv2104275896:84 , srv2137934854:84 , srv2144038819:84 , srv602208863:84 , srv906809155:84 , srv94889449:84 , srv1073947484:85 , srv130382898:85 , srv1494933506:85 , srv1508558671:85 , srv1630118433:85 , srv1773531095:85 , srv1948357614:85 , srv2091172568:85 , srv269168731:85 , srv466490222:85 , srv521089015:85 , srv641145052:85 , srv694183813:85 , srv945323714:85 , srv1034851246:86 , srv1171885443:86 , srv1471981274:86 , srv172056254:86 , srv1754055395:86 , srv1906222068:86 , srv378172006:86 , srv757452587:86 , srv922809520:86 , srv954127039:86 , srv995851835:86 , srv1282093968:87 , srv1310021704:87 , srv1370242462:87 , srv1553313638:87 , srv1645838413:87 , srv1657083072:87 , srv166004532:87 , srv1702806474:87 , srv173365499:87 , srv1983189741:87 , srv1992427617:87 , srv1995087550:87 , srv250770810:87 , srv312237669:87 , srv396598050:87 , srv408666192:87 , srv564048444:87 , srv579014209:87 , srv676991114:87 , srv746183153:87 , srv876735459:87 , srv1213718417:88 , srv1331128257:88 , srv1348931118:88 , srv1444215503:88 , srv1505620968:88 , srv1565179182:88 , srv1616753329:88 , srv1672845398:88 , srv1754549133:88 , srv81516030:88 , srv860239081:88 , srv962243138:88 , srv1434735727:89 , srv1718080046:89 , srv205103833:89 , srv2118384420:89 , srv58208467:89 , srv67333051:89 , srv755161500:89 , srv867467580:89 , srv1266999196:90 , srv1523029196:90 , srv1539195426:90 , srv1926023900:90 , srv383836623:90 , srv2060038489:91 , srv884462157:91 , srv2028925275:92 , srv536176949:92 , srv788962913:97 , srv197327035:444 } 2024-02-11 19:53:10,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 19:53:10,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1906222068=45, srv1754055395=40, srv1645838413=31, srv579014209=75, srv2028925275=54, srv1266999196=5, srv746183153=82, srv1325455818=10, srv94889449=95, srv564048444=74, srv1539195426=25, srv962243138=97, srv1918015934=46, srv1616753329=29, srv1630118433=30, srv1992427617=52, srv2060038489=56, srv1282093968=7, srv1471981274=19, srv1523029196=24, srv860239081=88, srv396598050=68, srv876735459=90, srv1553313638=26, srv166004532=33, srv173365499=39, srv1496492581=21, srv536176949=73, srv134614334=12, srv1693061383=35, srv788962913=85, srv1370988946=15, srv1983189741=50, srv2137934854=60, srv172056254=38, srv694183813=81, srv922809520=93, srv945323714=94, srv384131706=67, srv312237669=64, srv130382898=8, srv1702806474=36, srv1948357614=48, srv1853755032=44, srv58208467=76, srv1494933506=20, srv1718080046=37, srv1374678319=16, srv1348931118=13, srv521089015=72, srv1926023900=47, srv1657083072=32, srv2104275896=58, srv2118384420=59, srv676991114=80, srv197327035=49, srv1434735727=17, srv2091172568=57, srv2144038819=61, srv884462157=91, srv906809155=92, srv1331128257=11, srv1672845398=34, srv804694278=86, srv1565179182=27, srv1213718417=4, srv1505620968=22, srv205103833=55, srv422134787=70, srv995851835=98, srv1034851246=1, srv755161500=83, srv1572670677=28, srv1270681839=6, srv1826603363=43, srv269168731=63, srv602208863=77, srv954127039=96, srv1171885443=3, srv641145052=78, srv1754549133=41, srv378172006=65, srv757452587=84, srv1985252927=51, srv867467580=89, srv1444215503=18, srv408666192=69, srv67333051=79, srv466490222=71, srv996051775=99, srv1032013644=0, srv1370242462=14, srv1073947484=2, srv250770810=62, srv81516030=87, srv1773531095=42, srv383836623=66, srv1995087550=53, srv1310021704=9, srv1508558671=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 19:53:10,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 19:53:10,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 19:53:10,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 19:53:10,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 19:53:10,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 19:53:10,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 19:53:10,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 19:53:10,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 19:53:10,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 19:53:10,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 19:53:10,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 19:53:10,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 19:53:10,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 19:53:10,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 19:53:10,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 19:53:10,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 19:53:10,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 19:53:10,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 19:53:10,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 19:53:10,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 19:53:10,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 19:53:10,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 19:53:10,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 19:53:10,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 19:53:10,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 19:53:10,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 19:53:10,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 19:53:10,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 19:53:10,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 19:53:10,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 19:53:10,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 19:53:10,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 19:53:10,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 19:53:10,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 19:53:10,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 19:53:10,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 19:53:10,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 19:53:10,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 19:53:10,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 19:53:10,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 19:53:10,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 19:53:10,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 19:53:10,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 19:53:10,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 19:53:10,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 19:53:10,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 19:53:10,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 19:53:10,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 19:53:10,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 19:53:10,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 19:53:10,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 19:53:10,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 19:53:10,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 19:53:10,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 19:53:10,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 19:53:10,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 19:53:10,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 19:53:10,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 19:53:10,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 19:53:10,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 19:53:10,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 19:53:10,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 19:53:10,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 19:53:10,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 19:53:10,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 19:53:10,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 19:53:10,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 19:53:10,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 19:53:10,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 19:53:10,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 19:53:10,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 19:53:10,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 19:53:10,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 19:53:10,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 19:53:10,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 19:53:10,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 19:53:10,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 19:53:10,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 19:53:10,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 19:53:10,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 19:53:10,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 19:53:10,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 19:53:10,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 19:53:10,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 19:53:10,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 19:53:10,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 19:53:10,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 19:53:10,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 19:53:10,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 19:53:10,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 19:53:10,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 19:53:10,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 19:53:10,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 19:53:10,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 19:53:10,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 19:53:10,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 19:53:10,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 19:53:10,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 19:53:10,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 19:53:10,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 19:53:10,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 19:53:10,838 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039834424737766716, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0549502376658065, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039834424737766716, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17033333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23841666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 19:53:10,838 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-11 19:53:10,838 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17525038069148074, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039834424737766716, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0549502376658065, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039834424737766716, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17033333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23841666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-11 19:53:11,190 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 462 ms to try 5000 different iterations. Found a solution that moves 2632 regions; Going from a computed imbalance of 0.17525038069148074 to a new imbalance of 0.01549956146505786. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023386450150242002); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03583852049214947, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29244444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023386450150242002); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 19:53:11,210 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv788962913:76 , srv2028925275:79 , srv536176949:79 , srv1926023900:80 , srv1434735727:81 , srv2060038489:81 , srv1505620968:82 , srv1523029196:82 , srv1539195426:82 , srv205103833:82 , srv58208467:82 , srv676991114:82 , srv884462157:82 , srv1553313638:83 , srv1645838413:83 , srv1718080046:83 , srv1983189741:83 , srv1995087550:83 , srv2118384420:83 , srv383836623:83 , srv67333051:83 , srv755161500:83 , srv860239081:83 , srv876735459:83 , srv1213718417:84 , srv1331128257:84 , srv1471981274:84 , srv1565179182:84 , srv1657083072:84 , srv166004532:84 , srv1672845398:84 , srv1754549133:84 , srv396598050:84 , srv81516030:84 , srv867467580:84 , srv962243138:84 , srv995851835:84 , srv1034851246:85 , srv1266999196:85 , srv1282093968:85 , srv1370242462:85 , srv1444215503:85 , srv1494933506:85 , srv1616753329:85 , srv1630118433:85 , srv1702806474:85 , srv564048444:85 , srv746183153:85 , srv954127039:85 , srv130382898:86 , srv173365499:86 , srv1754055395:86 , srv1992427617:86 , srv2091172568:86 , srv466490222:86 , srv521089015:86 , srv579014209:86 , srv602208863:86 , srv757452587:86 , srv922809520:86 , srv1171885443:87 , srv1270681839:87 , srv1310021704:87 , srv1348931118:87 , srv172056254:87 , srv1773531095:87 , srv1906222068:87 , srv1948357614:87 , srv269168731:87 , srv312237669:87 , srv378172006:87 , srv408666192:87 , srv641145052:87 , srv694183813:87 , srv94889449:87 , srv134614334:88 , srv1508558671:88 , srv1572670677:88 , srv1693061383:88 , srv2104275896:88 , srv2144038819:88 , srv250770810:88 , srv422134787:88 , srv945323714:88 , srv1073947484:89 , srv1325455818:89 , srv1374678319:89 , srv1496492581:89 , srv1826603363:89 , srv1853755032:89 , srv1918015934:89 , srv2137934854:89 , srv1032013644:90 , srv1985252927:90 , srv384131706:90 , srv906809155:90 , srv996051775:90 , srv1370988946:91 , srv804694278:91 , srv197327035:538 } 2024-02-11 19:53:11,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 19:53:11,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1906222068=45, srv1754055395=40, srv1645838413=31, srv579014209=75, srv2028925275=54, srv1266999196=5, srv746183153=82, srv1325455818=10, srv94889449=95, srv564048444=74, srv1539195426=25, srv962243138=97, srv1918015934=46, srv1616753329=29, srv1630118433=30, srv1992427617=52, srv2060038489=56, srv1282093968=7, srv1471981274=19, srv1523029196=24, srv860239081=88, srv396598050=68, srv876735459=90, srv1553313638=26, srv166004532=33, srv173365499=39, srv1496492581=21, srv536176949=73, srv134614334=12, srv1693061383=35, srv788962913=85, srv1370988946=15, srv1983189741=50, srv2137934854=60, srv172056254=38, srv694183813=81, srv922809520=93, srv945323714=94, srv384131706=67, srv312237669=64, srv130382898=8, srv1702806474=36, srv1948357614=48, srv1853755032=44, srv58208467=76, srv1494933506=20, srv1718080046=37, srv1374678319=16, srv1348931118=13, srv521089015=72, srv1926023900=47, srv1657083072=32, srv2104275896=58, srv2118384420=59, srv676991114=80, srv197327035=49, srv1434735727=17, srv2091172568=57, srv2144038819=61, srv884462157=91, srv906809155=92, srv1331128257=11, srv1672845398=34, srv804694278=86, srv1565179182=27, srv1213718417=4, srv1505620968=22, srv205103833=55, srv422134787=70, srv995851835=98, srv1034851246=1, srv755161500=83, srv1572670677=28, srv1270681839=6, srv1826603363=43, srv269168731=63, srv602208863=77, srv954127039=96, srv1171885443=3, srv641145052=78, srv1754549133=41, srv378172006=65, srv757452587=84, srv1985252927=51, srv867467580=89, srv1444215503=18, srv408666192=69, srv67333051=79, srv466490222=71, srv996051775=99, srv1032013644=0, srv1370242462=14, srv1073947484=2, srv250770810=62, srv81516030=87, srv1773531095=42, srv383836623=66, srv1995087550=53, srv1310021704=9, srv1508558671=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 19:53:11,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 19:53:11,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 19:53:11,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 19:53:11,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 19:53:11,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 19:53:11,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 19:53:11,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 19:53:11,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 19:53:11,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 19:53:11,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 19:53:11,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 19:53:11,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 19:53:11,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 19:53:11,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 19:53:11,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 19:53:11,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 19:53:11,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 19:53:11,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 19:53:11,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 19:53:11,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 19:53:11,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 19:53:11,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 19:53:11,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 19:53:11,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 19:53:11,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 19:53:11,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 19:53:11,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 19:53:11,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 19:53:11,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 19:53:11,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 19:53:11,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 19:53:11,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 19:53:11,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 19:53:11,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 19:53:11,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 19:53:11,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 19:53:11,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 19:53:11,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 19:53:11,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 19:53:11,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 19:53:11,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 19:53:11,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 19:53:11,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 19:53:11,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 19:53:11,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 19:53:11,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 19:53:11,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 19:53:11,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 19:53:11,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 19:53:11,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 19:53:11,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 19:53:11,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 19:53:11,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 19:53:11,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 19:53:11,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 19:53:11,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 19:53:11,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 19:53:11,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 19:53:11,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 19:53:11,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 19:53:11,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 19:53:11,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 19:53:11,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 19:53:11,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 19:53:11,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 19:53:11,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 19:53:11,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 19:53:11,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 19:53:11,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 19:53:11,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 19:53:11,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 19:53:11,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 19:53:11,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 19:53:11,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 19:53:11,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 19:53:11,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 19:53:11,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 19:53:11,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 19:53:11,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 19:53:11,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 19:53:11,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 19:53:11,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 19:53:11,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 19:53:11,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 19:53:11,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 19:53:11,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 19:53:11,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 19:53:11,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 19:53:11,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 19:53:11,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 19:53:11,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 19:53:11,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 19:53:11,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 19:53:11,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 19:53:11,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 19:53:11,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 19:53:11,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 19:53:11,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 19:53:11,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 19:53:11,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 19:53:11,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 19:53:11,418 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023386450150242002); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03583852049214947, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023386450150242002); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 19:53:11,419 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-11 19:53:11,419 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015481131807152885, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023386450150242002); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03583852049214947, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023386450150242002); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-11 19:53:11,886 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 611 ms to try 5000 different iterations. Found a solution that moves 971 regions; Going from a computed imbalance of 0.015481131807152885 to a new imbalance of 0.0010103852551196953. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009575085098588816); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.016038673799259627); MoveCostFunction : (multiplier=7.0, imbalance=0.1078888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.009575085098588816); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 19:53:11,890 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1073947484:82 , srv906809155:82 , srv1266999196:83 , srv1348931118:83 , srv1370988946:83 , srv1508558671:83 , srv173365499:83 , srv1985252927:83 , srv250770810:83 , srv312237669:83 , srv408666192:83 , srv867467580:83 , srv1032013644:84 , srv1171885443:84 , srv1282093968:84 , srv1374678319:84 , srv1616753329:84 , srv172056254:84 , srv1773531095:84 , srv269168731:84 , srv579014209:84 , srv67333051:84 , srv788962913:84 , srv884462157:84 , srv945323714:84 , srv1310021704:85 , srv1325455818:85 , srv1331128257:85 , srv1370242462:85 , srv1444215503:85 , srv1523029196:85 , srv1539195426:85 , srv1565179182:85 , srv1672845398:85 , srv1693061383:85 , srv1702806474:85 , srv1718080046:85 , srv1754055395:85 , srv1754549133:85 , srv1826603363:85 , srv1906222068:85 , srv1918015934:85 , srv1992427617:85 , srv2104275896:85 , srv2118384420:85 , srv2137934854:85 , srv383836623:85 , srv466490222:85 , srv521089015:85 , srv755161500:85 , srv804694278:85 , srv81516030:85 , srv954127039:85 , srv962243138:85 , srv1034851246:86 , srv1213718417:86 , srv134614334:86 , srv1496492581:86 , srv1572670677:86 , srv1645838413:86 , srv166004532:86 , srv1853755032:86 , srv1926023900:86 , srv1948357614:86 , srv1995087550:86 , srv2028925275:86 , srv205103833:86 , srv2060038489:86 , srv2091172568:86 , srv2144038819:86 , srv378172006:86 , srv384131706:86 , srv396598050:86 , srv564048444:86 , srv58208467:86 , srv641145052:86 , srv694183813:86 , srv746183153:86 , srv757452587:86 , srv860239081:86 , srv922809520:86 , srv94889449:86 , srv996051775:86 , srv1270681839:87 , srv130382898:87 , srv1434735727:87 , srv1471981274:87 , srv1494933506:87 , srv1505620968:87 , srv1553313638:87 , srv1657083072:87 , srv1983189741:87 , srv422134787:87 , srv536176949:87 , srv602208863:87 , srv676991114:87 , srv995851835:87 , srv1630118433:88 , srv876735459:88 , srv197327035:561 } 2024-02-11 19:53:11,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 19:53:11,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1906222068=45, srv1754055395=40, srv1645838413=31, srv579014209=75, srv2028925275=54, srv1266999196=5, srv746183153=82, srv1325455818=10, srv94889449=95, srv564048444=74, srv1539195426=25, srv962243138=97, srv1918015934=46, srv1616753329=29, srv1630118433=30, srv1992427617=52, srv2060038489=56, srv1282093968=7, srv1471981274=19, srv1523029196=24, srv860239081=88, srv396598050=68, srv876735459=90, srv1553313638=26, srv166004532=33, srv173365499=39, srv1496492581=21, srv536176949=73, srv134614334=12, srv1693061383=35, srv788962913=85, srv1370988946=15, srv1983189741=50, srv2137934854=60, srv172056254=38, srv694183813=81, srv922809520=93, srv945323714=94, srv384131706=67, srv312237669=64, srv130382898=8, srv1702806474=36, srv1948357614=48, srv1853755032=44, srv58208467=76, srv1494933506=20, srv1718080046=37, srv1374678319=16, srv1348931118=13, srv521089015=72, srv1926023900=47, srv1657083072=32, srv2104275896=58, srv2118384420=59, srv676991114=80, srv197327035=49, srv1434735727=17, srv2091172568=57, srv2144038819=61, srv884462157=91, srv906809155=92, srv1331128257=11, srv1672845398=34, srv804694278=86, srv1565179182=27, srv1213718417=4, srv1505620968=22, srv205103833=55, srv422134787=70, srv995851835=98, srv1034851246=1, srv755161500=83, srv1572670677=28, srv1270681839=6, srv1826603363=43, srv269168731=63, srv602208863=77, srv954127039=96, srv1171885443=3, srv641145052=78, srv1754549133=41, srv378172006=65, srv757452587=84, srv1985252927=51, srv867467580=89, srv1444215503=18, srv408666192=69, srv67333051=79, srv466490222=71, srv996051775=99, srv1032013644=0, srv1370242462=14, srv1073947484=2, srv250770810=62, srv81516030=87, srv1773531095=42, srv383836623=66, srv1995087550=53, srv1310021704=9, srv1508558671=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 19:53:11,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 19:53:11,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 19:53:11,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 19:53:11,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 19:53:11,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 19:53:11,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 19:53:11,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 19:53:11,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 19:53:11,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 19:53:11,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 19:53:11,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 19:53:11,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 19:53:11,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 19:53:11,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 19:53:11,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 19:53:11,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 19:53:11,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 19:53:11,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 19:53:11,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 19:53:11,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 19:53:11,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 19:53:11,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 19:53:11,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 19:53:11,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 19:53:11,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 19:53:11,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 19:53:11,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 19:53:11,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 19:53:11,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 19:53:11,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 19:53:11,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 19:53:11,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 19:53:11,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 19:53:11,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 19:53:11,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 19:53:11,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 19:53:11,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 19:53:11,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 19:53:11,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 19:53:11,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 19:53:11,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 19:53:11,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 19:53:11,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 19:53:11,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 19:53:11,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 19:53:11,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 19:53:11,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 19:53:11,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 19:53:11,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 19:53:11,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 19:53:11,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 19:53:11,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 19:53:11,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 19:53:11,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 19:53:11,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 19:53:11,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 19:53:11,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 19:53:11,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 19:53:11,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 19:53:11,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 19:53:11,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 19:53:11,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 19:53:11,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 19:53:11,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 19:53:11,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 19:53:11,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 19:53:11,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 19:53:11,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 19:53:11,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 19:53:11,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 19:53:11,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 19:53:11,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 19:53:11,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 19:53:11,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 19:53:11,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 19:53:11,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 19:53:11,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 19:53:11,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 19:53:11,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 19:53:11,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 19:53:11,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 19:53:11,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 19:53:11,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 19:53:11,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 19:53:11,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 19:53:11,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 19:53:11,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 19:53:11,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 19:53:11,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 19:53:11,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 19:53:11,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 19:53:11,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 19:53:11,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 19:53:11,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 19:53:11,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 19:53:11,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 19:53:11,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 19:53:11,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 19:53:11,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 19:53:11,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 19:53:11,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 19:53:12,075 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009575085098588816); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.016038673799259627); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.009575085098588816); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 19:53:12,075 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-11 19:53:12,076 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.001003586167799888, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009575085098588816); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.016038673799259627); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.009575085098588816); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-11 19:53:12,315 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 369 ms to try 5000 different iterations. Found a solution that moves 306 regions; Going from a computed imbalance of 0.001003586167799888 to a new imbalance of 4.281701923947311E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.034, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 19:53:12,316 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1310021704:83 , srv1906222068:83 , srv1992427617:83 , srv378172006:83 , srv757452587:83 , srv922809520:83 , srv996051775:83 , srv1213718417:84 , srv1266999196:84 , srv130382898:84 , srv1348931118:84 , srv1444215503:84 , srv1496492581:84 , srv1630118433:84 , srv1657083072:84 , srv1718080046:84 , srv1948357614:84 , srv205103833:84 , srv2060038489:84 , srv2144038819:84 , srv250770810:84 , srv383836623:84 , srv408666192:84 , srv422134787:84 , srv536176949:84 , srv564048444:84 , srv694183813:84 , srv746183153:84 , srv804694278:84 , srv876735459:84 , srv1032013644:85 , srv1034851246:85 , srv1073947484:85 , srv1270681839:85 , srv1325455818:85 , srv1331128257:85 , srv134614334:85 , srv1370242462:85 , srv1370988946:85 , srv1434735727:85 , srv1471981274:85 , srv1494933506:85 , srv1505620968:85 , srv1523029196:85 , srv1539195426:85 , srv1553313638:85 , srv1572670677:85 , srv1616753329:85 , srv1645838413:85 , srv166004532:85 , srv1672845398:85 , srv1702806474:85 , srv172056254:85 , srv1754055395:85 , srv1754549133:85 , srv1826603363:85 , srv1853755032:85 , srv1918015934:85 , srv1983189741:85 , srv2028925275:85 , srv2091172568:85 , srv2104275896:85 , srv2118384420:85 , srv2137934854:85 , srv269168731:85 , srv312237669:85 , srv384131706:85 , srv396598050:85 , srv58208467:85 , srv602208863:85 , srv641145052:85 , srv755161500:85 , srv788962913:85 , srv860239081:85 , srv884462157:85 , srv94889449:85 , srv962243138:85 , srv995851835:85 , srv1171885443:86 , srv1374678319:86 , srv1508558671:86 , srv1565179182:86 , srv1693061383:86 , srv173365499:86 , srv1773531095:86 , srv1926023900:86 , srv1995087550:86 , srv466490222:86 , srv521089015:86 , srv579014209:86 , srv67333051:86 , srv676991114:86 , srv81516030:86 , srv867467580:86 , srv906809155:86 , srv945323714:86 , srv954127039:86 , srv1282093968:87 , srv1985252927:87 , srv197327035:599 } 2024-02-11 19:53:12,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 19:53:12,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1906222068=45, srv1754055395=40, srv1645838413=31, srv579014209=75, srv2028925275=54, srv1266999196=5, srv746183153=82, srv1325455818=10, srv94889449=95, srv564048444=74, srv1539195426=25, srv962243138=97, srv1918015934=46, srv1616753329=29, srv1630118433=30, srv1992427617=52, srv2060038489=56, srv1282093968=7, srv1471981274=19, srv1523029196=24, srv860239081=88, srv396598050=68, srv876735459=90, srv1553313638=26, srv166004532=33, srv173365499=39, srv1496492581=21, srv536176949=73, srv134614334=12, srv1693061383=35, srv788962913=85, srv1370988946=15, srv1983189741=50, srv2137934854=60, srv172056254=38, srv694183813=81, srv922809520=93, srv945323714=94, srv384131706=67, srv312237669=64, srv130382898=8, srv1702806474=36, srv1948357614=48, srv1853755032=44, srv58208467=76, srv1494933506=20, srv1718080046=37, srv1374678319=16, srv1348931118=13, srv521089015=72, srv1926023900=47, srv1657083072=32, srv2104275896=58, srv2118384420=59, srv676991114=80, srv197327035=49, srv1434735727=17, srv2091172568=57, srv2144038819=61, srv884462157=91, srv906809155=92, srv1331128257=11, srv1672845398=34, srv804694278=86, srv1565179182=27, srv1213718417=4, srv1505620968=22, srv205103833=55, srv422134787=70, srv995851835=98, srv1034851246=1, srv755161500=83, srv1572670677=28, srv1270681839=6, srv1826603363=43, srv269168731=63, srv602208863=77, srv954127039=96, srv1171885443=3, srv641145052=78, srv1754549133=41, srv378172006=65, srv757452587=84, srv1985252927=51, srv867467580=89, srv1444215503=18, srv408666192=69, srv67333051=79, srv466490222=71, srv996051775=99, srv1032013644=0, srv1370242462=14, srv1073947484=2, srv250770810=62, srv81516030=87, srv1773531095=42, srv383836623=66, srv1995087550=53, srv1310021704=9, srv1508558671=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 19:53:12,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 19:53:12,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 19:53:12,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 19:53:12,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 19:53:12,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 19:53:12,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 19:53:12,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 19:53:12,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 19:53:12,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 19:53:12,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 19:53:12,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 19:53:12,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 19:53:12,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 19:53:12,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 19:53:12,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 19:53:12,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 19:53:12,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 19:53:12,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 19:53:12,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 19:53:12,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 19:53:12,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 19:53:12,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 19:53:12,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 19:53:12,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 19:53:12,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 19:53:12,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 19:53:12,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 19:53:12,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 19:53:12,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 19:53:12,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 19:53:12,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 19:53:12,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 19:53:12,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 19:53:12,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 19:53:12,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 19:53:12,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 19:53:12,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 19:53:12,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 19:53:12,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 19:53:12,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 19:53:12,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 19:53:12,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 19:53:12,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 19:53:12,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 19:53:12,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 19:53:12,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 19:53:12,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 19:53:12,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 19:53:12,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 19:53:12,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 19:53:12,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 19:53:12,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 19:53:12,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 19:53:12,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 19:53:12,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 19:53:12,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 19:53:12,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 19:53:12,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 19:53:12,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 19:53:12,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 19:53:12,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 19:53:12,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 19:53:12,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 19:53:12,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 19:53:12,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 19:53:12,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 19:53:12,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 19:53:12,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 19:53:12,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 19:53:12,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 19:53:12,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 19:53:12,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 19:53:12,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 19:53:12,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 19:53:12,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 19:53:12,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 19:53:12,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 19:53:12,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 19:53:12,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 19:53:12,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 19:53:12,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 19:53:12,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 19:53:12,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 19:53:12,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 19:53:12,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 19:53:12,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 19:53:12,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 19:53:12,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 19:53:12,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 19:53:12,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 19:53:12,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 19:53:12,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 19:53:12,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 19:53:12,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 19:53:12,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 19:53:12,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 19:53:12,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 19:53:12,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 19:53:12,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 19:53:12,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 19:53:12,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 19:53:12,437 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 19:53:12,437 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-02-11 19:53:12,438 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.067436144352976E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-02-11 19:53:12,673 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 329 ms to try 5000 different iterations. Found a solution that moves 32 regions; Going from a computed imbalance of 4.067436144352976E-5 to a new imbalance of 6.256147528846628E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0013400504203456161); MoveCostFunction : (multiplier=7.0, imbalance=0.0035555555555555557); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 19:53:12,673 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1171885443:83 , srv1282093968:83 , srv1374678319:83 , srv1565179182:83 , srv1985252927:83 , srv2137934854:83 , srv579014209:83 , srv641145052:83 , srv81516030:83 , srv945323714:83 , srv1032013644:84 , srv1034851246:84 , srv1213718417:84 , srv1266999196:84 , srv1270681839:84 , srv130382898:84 , srv1310021704:84 , srv1325455818:84 , srv1331128257:84 , srv134614334:84 , srv1348931118:84 , srv1370242462:84 , srv1370988946:84 , srv1434735727:84 , srv1444215503:84 , srv1471981274:84 , srv1494933506:84 , srv1496492581:84 , srv1505620968:84 , srv1508558671:84 , srv1523029196:84 , srv1539195426:84 , srv1553313638:84 , srv1572670677:84 , srv1616753329:84 , srv1630118433:84 , srv1657083072:84 , srv166004532:84 , srv1672845398:84 , srv1693061383:84 , srv1702806474:84 , srv172056254:84 , srv173365499:84 , srv1754055395:84 , srv1754549133:84 , srv1773531095:84 , srv1826603363:84 , srv1853755032:84 , srv1918015934:84 , srv1926023900:84 , srv1948357614:84 , srv1983189741:84 , srv1995087550:84 , srv2028925275:84 , srv2060038489:84 , srv2091172568:84 , srv2104275896:84 , srv2118384420:84 , srv2144038819:84 , srv250770810:84 , srv312237669:84 , srv378172006:84 , srv383836623:84 , srv384131706:84 , srv396598050:84 , srv422134787:84 , srv466490222:84 , srv521089015:84 , srv536176949:84 , srv564048444:84 , srv58208467:84 , srv602208863:84 , srv67333051:84 , srv676991114:84 , srv694183813:84 , srv746183153:84 , srv755161500:84 , srv788962913:84 , srv804694278:84 , srv860239081:84 , srv867467580:84 , srv876735459:84 , srv884462157:84 , srv906809155:84 , srv94889449:84 , srv954127039:84 , srv962243138:84 , srv995851835:84 , srv996051775:84 , srv1073947484:85 , srv1645838413:85 , srv1718080046:85 , srv1906222068:85 , srv1992427617:85 , srv205103833:85 , srv269168731:85 , srv408666192:85 , srv757452587:85 , srv922809520:85 , srv197327035:684 } 2024-02-11 19:53:12,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 19:53:12,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1906222068=45, srv1754055395=40, srv1645838413=31, srv579014209=75, srv2028925275=54, srv1266999196=5, srv746183153=82, srv1325455818=10, srv94889449=95, srv564048444=74, srv1539195426=25, srv962243138=97, srv1918015934=46, srv1616753329=29, srv1630118433=30, srv1992427617=52, srv2060038489=56, srv1282093968=7, srv1471981274=19, srv1523029196=24, srv860239081=88, srv396598050=68, srv876735459=90, srv1553313638=26, srv166004532=33, srv173365499=39, srv1496492581=21, srv536176949=73, srv134614334=12, srv1693061383=35, srv788962913=85, srv1370988946=15, srv1983189741=50, srv2137934854=60, srv172056254=38, srv694183813=81, srv922809520=93, srv945323714=94, srv384131706=67, srv312237669=64, srv130382898=8, srv1702806474=36, srv1948357614=48, srv1853755032=44, srv58208467=76, srv1494933506=20, srv1718080046=37, srv1374678319=16, srv1348931118=13, srv521089015=72, srv1926023900=47, srv1657083072=32, srv2104275896=58, srv2118384420=59, srv676991114=80, srv197327035=49, srv1434735727=17, srv2091172568=57, srv2144038819=61, srv884462157=91, srv906809155=92, srv1331128257=11, srv1672845398=34, srv804694278=86, srv1565179182=27, srv1213718417=4, srv1505620968=22, srv205103833=55, srv422134787=70, srv995851835=98, srv1034851246=1, srv755161500=83, srv1572670677=28, srv1270681839=6, srv1826603363=43, srv269168731=63, srv602208863=77, srv954127039=96, srv1171885443=3, srv641145052=78, srv1754549133=41, srv378172006=65, srv757452587=84, srv1985252927=51, srv867467580=89, srv1444215503=18, srv408666192=69, srv67333051=79, srv466490222=71, srv996051775=99, srv1032013644=0, srv1370242462=14, srv1073947484=2, srv250770810=62, srv81516030=87, srv1773531095=42, srv383836623=66, srv1995087550=53, srv1310021704=9, srv1508558671=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 19:53:12,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 19:53:12,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 19:53:12,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 19:53:12,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 19:53:12,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 19:53:12,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 19:53:12,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 19:53:12,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 19:53:12,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 19:53:12,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 19:53:12,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 19:53:12,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 19:53:12,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 19:53:12,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 19:53:12,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 19:53:12,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 19:53:12,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 19:53:12,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 19:53:12,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 19:53:12,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 19:53:12,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 19:53:12,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 19:53:12,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 19:53:12,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 19:53:12,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 19:53:12,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 19:53:12,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 19:53:12,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 19:53:12,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 19:53:12,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 19:53:12,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-02-11 19:53:12,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-02-11 19:53:12,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-02-11 19:53:12,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-02-11 19:53:12,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-02-11 19:53:12,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-02-11 19:53:12,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-02-11 19:53:12,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-02-11 19:53:12,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-02-11 19:53:12,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-02-11 19:53:12,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-02-11 19:53:12,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-02-11 19:53:12,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-02-11 19:53:12,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-02-11 19:53:12,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-02-11 19:53:12,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-02-11 19:53:12,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-02-11 19:53:12,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-02-11 19:53:12,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-02-11 19:53:12,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-02-11 19:53:12,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-02-11 19:53:12,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-02-11 19:53:12,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-02-11 19:53:12,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-02-11 19:53:12,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-02-11 19:53:12,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-02-11 19:53:12,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-02-11 19:53:12,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-02-11 19:53:12,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-02-11 19:53:12,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-02-11 19:53:12,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-02-11 19:53:12,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-02-11 19:53:12,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-02-11 19:53:12,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-02-11 19:53:12,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-02-11 19:53:12,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-02-11 19:53:12,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-02-11 19:53:12,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-02-11 19:53:12,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-02-11 19:53:12,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-02-11 19:53:12,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-02-11 19:53:12,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-02-11 19:53:12,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-02-11 19:53:12,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-02-11 19:53:12,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-02-11 19:53:12,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-02-11 19:53:12,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-02-11 19:53:12,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-02-11 19:53:12,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-02-11 19:53:12,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-02-11 19:53:12,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-02-11 19:53:12,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-02-11 19:53:12,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-02-11 19:53:12,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-02-11 19:53:12,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-02-11 19:53:12,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-02-11 19:53:12,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-02-11 19:53:12,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-02-11 19:53:12,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-02-11 19:53:12,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-02-11 19:53:12,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-02-11 19:53:12,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-02-11 19:53:12,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-02-11 19:53:12,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-02-11 19:53:12,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-02-11 19:53:12,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-02-11 19:53:12,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-02-11 19:53:12,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-02-11 19:53:12,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-02-11 19:53:12,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-02-11 19:53:12,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-02-11 19:53:12,812 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.032078739728369E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0013400504203456161); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 19:53:12,812 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1171885443:83 , srv1282093968:83 , srv1374678319:83 , srv1565179182:83 , srv1985252927:83 , srv2137934854:83 , srv579014209:83 , srv641145052:83 , srv81516030:83 , srv945323714:83 , srv1032013644:84 , srv1034851246:84 , srv1213718417:84 , srv1266999196:84 , srv1270681839:84 , srv130382898:84 , srv1310021704:84 , srv1325455818:84 , srv1331128257:84 , srv134614334:84 , srv1348931118:84 , srv1370242462:84 , srv1370988946:84 , srv1434735727:84 , srv1444215503:84 , srv1471981274:84 , srv1494933506:84 , srv1496492581:84 , srv1505620968:84 , srv1508558671:84 , srv1523029196:84 , srv1539195426:84 , srv1553313638:84 , srv1572670677:84 , srv1616753329:84 , srv1630118433:84 , srv1657083072:84 , srv166004532:84 , srv1672845398:84 , srv1693061383:84 , srv1702806474:84 , srv172056254:84 , srv173365499:84 , srv1754055395:84 , srv1754549133:84 , srv1773531095:84 , srv1826603363:84 , srv1853755032:84 , srv1918015934:84 , srv1926023900:84 , srv1948357614:84 , srv1983189741:84 , srv1995087550:84 , srv2028925275:84 , srv2060038489:84 , srv2091172568:84 , srv2104275896:84 , srv2118384420:84 , srv2144038819:84 , srv250770810:84 , srv312237669:84 , srv378172006:84 , srv383836623:84 , srv384131706:84 , srv396598050:84 , srv422134787:84 , srv466490222:84 , srv521089015:84 , srv536176949:84 , srv564048444:84 , srv58208467:84 , srv602208863:84 , srv67333051:84 , srv676991114:84 , srv694183813:84 , srv746183153:84 , srv755161500:84 , srv788962913:84 , srv804694278:84 , srv860239081:84 , srv867467580:84 , srv876735459:84 , srv884462157:84 , srv906809155:84 , srv94889449:84 , srv954127039:84 , srv962243138:84 , srv995851835:84 , srv996051775:84 , srv1073947484:85 , srv1645838413:85 , srv1718080046:85 , srv1906222068:85 , srv1992427617:85 , srv205103833:85 , srv269168731:85 , srv408666192:85 , srv757452587:85 , srv922809520:85 , srv197327035:684 } 2024-02-11 19:53:12,842 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=270 (was 270), ProcessCount=170 (was 170), AvailableMemoryMB=2375 (was 3160) 2024-02-11 19:53:12,849 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=270, ProcessCount=170, AvailableMemoryMB=2375 2024-02-11 19:53:12,849 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-02-11 19:53:12,850 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-02-11 19:53:12,860 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1043520221:84 , srv1056874077:84 , srv1073881390:84 , srv1244256380:84 , srv1269306025:84 , srv1282631181:84 , srv1366671154:84 , srv1420700542:84 , srv143302888:84 , srv1433840724:84 , srv1499293125:84 , srv1513851869:84 , srv1548132032:84 , srv1729571356:84 , srv1827998380:84 , srv1896112496:84 , srv2088834213:84 , srv2115926057:84 , srv256421155:84 , srv26977469:84 , srv343154022:84 , srv390501587:84 , srv391976601:84 , srv511869166:84 , srv518471434:84 , srv60392128:84 , srv740298277:84 , srv815648386:84 , srv953749171:84 , srv776316019:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-02-11 19:53:12,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 19:53:12,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1513851869=11, srv953749171=29, srv776316019=27, srv143302888=8, srv1073881390=2, srv26977469=19, srv740298277=26, srv1043520221=0, srv1269306025=4, srv511869166=23, srv256421155=18, srv1420700542=7, srv1548132032=12, srv343154022=20, srv1499293125=10, srv391976601=22, srv1827998380=14, srv390501587=21, srv815648386=28, srv1433840724=9, srv1056874077=1, srv2088834213=16, srv60392128=25, srv1244256380=3, srv518471434=24, srv1729571356=13, srv1896112496=15, srv2115926057=17, srv1282631181=5, srv1366671154=6} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 19:53:12,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 19:53:12,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 19:53:12,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 19:53:12,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 19:53:12,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 19:53:12,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 19:53:12,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 19:53:12,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 19:53:12,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 19:53:12,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 19:53:12,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 19:53:12,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 19:53:12,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 19:53:12,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 19:53:12,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 19:53:12,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 19:53:12,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 19:53:12,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 19:53:12,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 19:53:12,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 19:53:12,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 19:53:12,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 19:53:12,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 19:53:12,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 19:53:12,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 19:53:12,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 19:53:12,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 19:53:12,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 19:53:12,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 19:53:12,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 19:53:12,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-02-11 19:53:12,904 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.098692933727173, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 19:53:12,904 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912504021475833, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.098692933727173, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-02-11 19:55:12,886 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6417899 different iterations. Found a solution that moves 1851 regions; Going from a computed imbalance of 0.9912504021475833 to a new imbalance of 4.320326340186438E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6855555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 19:55:12,891 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1043520221:90 , srv1056874077:90 , srv1073881390:90 , srv1244256380:90 , srv1269306025:90 , srv1282631181:90 , srv1366671154:90 , srv1420700542:90 , srv143302888:90 , srv1433840724:90 , srv1499293125:90 , srv1513851869:90 , srv1548132032:90 , srv1729571356:90 , srv1827998380:90 , srv1896112496:90 , srv2088834213:90 , srv2115926057:90 , srv256421155:90 , srv26977469:90 , srv343154022:90 , srv390501587:90 , srv391976601:90 , srv511869166:90 , srv518471434:90 , srv60392128:90 , srv740298277:90 , srv776316019:90 , srv815648386:90 , srv953749171:90 } 2024-02-11 19:55:12,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-02-11 19:55:12,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1513851869=11, srv953749171=29, srv776316019=27, srv143302888=8, srv1073881390=2, srv26977469=19, srv740298277=26, srv1043520221=0, srv1269306025=4, srv511869166=23, srv256421155=18, srv1420700542=7, srv1548132032=12, srv343154022=20, srv1499293125=10, srv391976601=22, srv1827998380=14, srv390501587=21, srv815648386=28, srv1433840724=9, srv1056874077=1, srv2088834213=16, srv60392128=25, srv1244256380=3, srv518471434=24, srv1729571356=13, srv1896112496=15, srv2115926057=17, srv1282631181=5, srv1366671154=6} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-02-11 19:55:12,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-02-11 19:55:12,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-02-11 19:55:12,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-02-11 19:55:12,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-02-11 19:55:12,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-02-11 19:55:12,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-02-11 19:55:12,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-02-11 19:55:12,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-02-11 19:55:12,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-02-11 19:55:12,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-02-11 19:55:12,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-02-11 19:55:12,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-02-11 19:55:12,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-02-11 19:55:12,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-02-11 19:55:12,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-02-11 19:55:12,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-02-11 19:55:12,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-02-11 19:55:12,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-02-11 19:55:12,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-02-11 19:55:12,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-02-11 19:55:12,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-02-11 19:55:12,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-02-11 19:55:12,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-02-11 19:55:12,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-02-11 19:55:12,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-02-11 19:55:12,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-02-11 19:55:12,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-02-11 19:55:12,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-02-11 19:55:12,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-02-11 19:55:12,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-02-11 19:55:12,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-02-11 19:55:12,908 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-02-11 19:55:12,909 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1043520221:90 , srv1056874077:90 , srv1073881390:90 , srv1244256380:90 , srv1269306025:90 , srv1282631181:90 , srv1366671154:90 , srv1420700542:90 , srv143302888:90 , srv1433840724:90 , srv1499293125:90 , srv1513851869:90 , srv1548132032:90 , srv1729571356:90 , srv1827998380:90 , srv1896112496:90 , srv2088834213:90 , srv2115926057:90 , srv256421155:90 , srv26977469:90 , srv343154022:90 , srv390501587:90 , srv391976601:90 , srv511869166:90 , srv518471434:90 , srv60392128:90 , srv740298277:90 , srv776316019:90 , srv815648386:90 , srv953749171:90 } 2024-02-11 19:55:12,920 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=189 (was 270), ProcessCount=167 (was 170), AvailableMemoryMB=2015 (was 2375)