2023-12-25 02:53:15,117 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-12-25 02:53:15,459 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-12-25 02:53:15,590 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-12-25 02:53:15,651 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-25 02:53:15,659 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-25 02:53:15,676 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=284, ProcessCount=172, AvailableMemoryMB=11781 2023-12-25 02:53:15,678 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-25 02:53:15,678 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-25 02:53:15,813 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1010718364:84 , srv1035759288:84 , srv1039885596:84 , srv1042954044:84 , srv1044459326:84 , srv1059833356:84 , srv1120220051:84 , srv1124831258:84 , srv1125708709:84 , srv1138712124:84 , srv1185136671:84 , srv1190960103:84 , srv1220625257:84 , srv1227281076:84 , srv1229688104:84 , srv1260853910:84 , srv1264636558:84 , srv1289596928:84 , srv1297083462:84 , srv1327555290:84 , srv135315108:84 , srv1358884082:84 , srv1397433796:84 , srv140449034:84 , srv1424267197:84 , srv1436648944:84 , srv1474914701:84 , srv1475932371:84 , srv1513472637:84 , srv1513889077:84 , srv153421471:84 , srv1571168493:84 , srv158946621:84 , srv1589949938:84 , srv1590829361:84 , srv1594345136:84 , srv1595527166:84 , srv1617260499:84 , srv1625315554:84 , srv1660404233:84 , srv1664415141:84 , srv1690311052:84 , srv1727659229:84 , srv1748244579:84 , srv1750818822:84 , srv1770737853:84 , srv1775073491:84 , srv178784677:84 , srv1800691434:84 , srv1831884162:84 , srv1840490073:84 , srv1877540077:84 , srv1929793166:84 , srv1932079391:84 , srv1957164435:84 , srv1967166138:84 , srv1996401046:84 , srv2016153709:84 , srv2036544196:84 , srv2039496243:84 , srv206060290:84 , srv2109479112:84 , srv2112147671:84 , srv2115325635:84 , srv2119091690:84 , srv2136186693:84 , srv273903558:84 , srv2762825:84 , srv291513141:84 , srv308944613:84 , srv312395502:84 , srv332213759:84 , srv339172710:84 , srv412273032:84 , srv426568388:84 , srv437620312:84 , srv440071855:84 , srv453702029:84 , srv472041493:84 , srv49643950:84 , srv528411091:84 , srv580544388:84 , srv598935079:84 , srv622514671:84 , srv675402669:84 , srv761616732:84 , srv777123019:84 , srv796412229:84 , srv824822545:84 , srv848384040:84 , srv850337292:84 , srv865817248:84 , srv896165483:84 , srv906731712:84 , srv926113368:84 , srv927115734:84 , srv928930748:84 , srv953282164:84 , srv956051400:84 , srv476783668:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-12-25 02:53:15,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-25 02:53:15,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv761616732=86, srv926113368=95, srv273903558=66, srv1327555290=19, srv2112147671=62, srv308944613=69, srv135315108=20, srv906731712=94, srv2136186693=65, srv332213759=71, srv437620312=75, srv1595527166=36, srv1664415141=40, srv1260853910=15, srv178784677=47, srv850337292=91, srv1475932371=27, srv1220625257=12, srv1831884162=49, srv896165483=93, srv1120220051=6, srv1042954044=3, srv1125708709=8, srv1967166138=55, srv1397433796=22, srv1010718364=0, srv1750818822=44, srv1932079391=53, srv1044459326=4, srv2039496243=59, srv956051400=99, srv1035759288=1, srv848384040=90, srv865817248=92, srv1800691434=48, srv2115325635=63, srv1775073491=46, srv412273032=73, srv153421471=30, srv1513889077=29, srv291513141=68, srv777123019=87, srv2762825=67, srv1594345136=35, srv1264636558=16, srv796412229=88, srv2036544196=58, srv2119091690=64, srv675402669=85, srv1660404233=39, srv1690311052=41, srv440071855=76, srv953282164=98, srv1957164435=54, srv824822545=89, srv1727659229=42, srv598935079=83, srv1289596928=17, srv580544388=82, srv1877540077=51, srv1436648944=25, srv1424267197=24, srv472041493=78, srv1190960103=11, srv1124831258=7, srv1297083462=18, srv1185136671=10, srv1625315554=38, srv339172710=72, srv2016153709=57, srv1617260499=37, srv1229688104=14, srv1748244579=43, srv1996401046=56, srv1513472637=28, srv426568388=74, srv622514671=84, srv1571168493=31, srv1590829361=34, srv2109479112=61, srv1039885596=2, srv1474914701=26, srv158946621=32, srv476783668=79, srv927115734=96, srv206060290=60, srv1227281076=13, srv528411091=81, srv1589949938=33, srv453702029=77, srv1770737853=45, srv1929793166=52, srv1059833356=5, srv49643950=80, srv1138712124=9, srv312395502=70, srv928930748=97, srv1358884082=21, srv140449034=23, srv1840490073=50} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-25 02:53:15,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-25 02:53:15,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-25 02:53:15,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-25 02:53:15,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-25 02:53:15,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-25 02:53:15,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-25 02:53:15,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-25 02:53:15,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-25 02:53:15,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-25 02:53:15,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-25 02:53:15,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-25 02:53:15,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-25 02:53:15,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-25 02:53:15,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-25 02:53:15,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-25 02:53:15,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-25 02:53:15,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-25 02:53:15,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-25 02:53:15,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-25 02:53:15,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-25 02:53:15,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-25 02:53:15,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-25 02:53:15,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-25 02:53:15,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-25 02:53:15,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-25 02:53:15,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-25 02:53:15,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-25 02:53:15,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-25 02:53:15,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-25 02:53:15,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-25 02:53:15,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-25 02:53:15,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-25 02:53:15,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-25 02:53:15,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-25 02:53:15,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-25 02:53:15,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-25 02:53:15,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-25 02:53:15,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-25 02:53:15,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-25 02:53:15,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-25 02:53:15,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-25 02:53:15,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-25 02:53:15,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-25 02:53:15,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-25 02:53:15,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-25 02:53:15,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-25 02:53:15,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-25 02:53:15,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-25 02:53:15,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-25 02:53:15,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-25 02:53:15,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-25 02:53:15,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-25 02:53:15,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-25 02:53:15,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-25 02:53:15,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-25 02:53:15,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-25 02:53:15,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-25 02:53:15,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-25 02:53:15,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-25 02:53:15,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-25 02:53:15,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-25 02:53:15,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-25 02:53:15,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-25 02:53:15,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-25 02:53:15,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-25 02:53:15,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-25 02:53:15,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-25 02:53:15,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-25 02:53:15,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-25 02:53:15,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-25 02:53:15,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-25 02:53:15,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-25 02:53:15,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-25 02:53:15,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-25 02:53:15,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-25 02:53:15,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-25 02:53:15,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-25 02:53:15,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-25 02:53:15,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-25 02:53:15,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-25 02:53:15,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-25 02:53:15,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-25 02:53:15,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-25 02:53:15,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-25 02:53:15,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-25 02:53:15,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-25 02:53:15,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-25 02:53:15,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-25 02:53:15,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-25 02:53:15,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-25 02:53:15,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-25 02:53:15,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-25 02:53:15,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-25 02:53:15,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-25 02:53:15,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-25 02:53:15,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-25 02:53:15,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-25 02:53:15,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-25 02:53:15,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-25 02:53:15,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-25 02:53:15,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-25 02:53:16,374 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-25 02:53:16,374 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-25 02:53:16,375 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-25 02:53:17,094 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1107 ms to try 5000 different iterations. Found a solution that moves 4694 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1769245932794467. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04076889140689198, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06024827048572235, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5215555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04076889140689198, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17208333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23883333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-25 02:53:17,104 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1840490073:79 , srv1059833356:80 , srv1590829361:80 , srv598935079:80 , srv1770737853:81 , srv291513141:81 , srv896165483:81 , srv1138712124:82 , srv158946621:82 , srv848384040:82 , srv1039885596:83 , srv1264636558:83 , srv1397433796:83 , srv339172710:83 , srv528411091:83 , srv865817248:83 , srv1010718364:84 , srv1044459326:84 , srv1185136671:84 , srv1227281076:84 , srv153421471:84 , srv1690311052:84 , srv1957164435:84 , srv437620312:84 , srv622514671:84 , srv675402669:84 , srv927115734:84 , srv1297083462:85 , srv1474914701:85 , srv1800691434:85 , srv1877540077:85 , srv1996401046:85 , srv206060290:85 , srv440071855:85 , srv580544388:85 , srv824822545:85 , srv1220625257:86 , srv1327555290:86 , srv1424267197:86 , srv1589949938:86 , srv1594345136:86 , srv1664415141:86 , srv178784677:86 , srv1929793166:86 , srv2109479112:86 , srv2115325635:86 , srv412273032:86 , srv426568388:86 , srv472041493:86 , srv953282164:86 , srv1190960103:87 , srv1260853910:87 , srv1358884082:87 , srv140449034:87 , srv1436648944:87 , srv1475932371:87 , srv1513889077:87 , srv1571168493:87 , srv1617260499:87 , srv1660404233:87 , srv1967166138:87 , srv2119091690:87 , srv308944613:87 , srv796412229:87 , srv928930748:87 , srv1035759288:88 , srv1125708709:88 , srv1229688104:88 , srv1595527166:88 , srv1625315554:88 , srv1775073491:88 , srv1932079391:88 , srv2016153709:88 , srv273903558:88 , srv312395502:88 , srv332213759:88 , srv761616732:88 , srv850337292:88 , srv906731712:88 , srv1042954044:89 , srv1120220051:89 , srv1124831258:89 , srv1750818822:89 , srv2036544196:89 , srv2136186693:89 , srv453702029:89 , srv1513472637:90 , srv2039496243:90 , srv49643950:90 , srv1727659229:91 , srv1831884162:91 , srv2112147671:91 , srv777123019:91 , srv926113368:91 , srv135315108:92 , srv1748244579:92 , srv1289596928:93 , srv956051400:93 , srv2762825:94 , srv476783668:452 } 2023-12-25 02:53:17,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-25 02:53:17,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv761616732=86, srv926113368=95, srv273903558=66, srv1327555290=19, srv2112147671=62, srv308944613=69, srv135315108=20, srv906731712=94, srv2136186693=65, srv332213759=71, srv437620312=75, srv1595527166=36, srv1664415141=40, srv1260853910=15, srv178784677=47, srv850337292=91, srv1475932371=27, srv1220625257=12, srv1831884162=49, srv896165483=93, srv1120220051=6, srv1042954044=3, srv1125708709=8, srv1967166138=55, srv1397433796=22, srv1010718364=0, srv1750818822=44, srv1932079391=53, srv1044459326=4, srv2039496243=59, srv956051400=99, srv1035759288=1, srv848384040=90, srv865817248=92, srv1800691434=48, srv2115325635=63, srv1775073491=46, srv412273032=73, srv153421471=30, srv1513889077=29, srv291513141=68, srv777123019=87, srv2762825=67, srv1594345136=35, srv1264636558=16, srv796412229=88, srv2036544196=58, srv2119091690=64, srv675402669=85, srv1660404233=39, srv1690311052=41, srv440071855=76, srv953282164=98, srv1957164435=54, srv824822545=89, srv1727659229=42, srv598935079=83, srv1289596928=17, srv580544388=82, srv1877540077=51, srv1436648944=25, srv1424267197=24, srv472041493=78, srv1190960103=11, srv1124831258=7, srv1297083462=18, srv1185136671=10, srv1625315554=38, srv339172710=72, srv2016153709=57, srv1617260499=37, srv1229688104=14, srv1748244579=43, srv1996401046=56, srv1513472637=28, srv426568388=74, srv622514671=84, srv1571168493=31, srv1590829361=34, srv2109479112=61, srv1039885596=2, srv1474914701=26, srv158946621=32, srv476783668=79, srv927115734=96, srv206060290=60, srv1227281076=13, srv528411091=81, srv1589949938=33, srv453702029=77, srv1770737853=45, srv1929793166=52, srv1059833356=5, srv49643950=80, srv1138712124=9, srv312395502=70, srv928930748=97, srv1358884082=21, srv140449034=23, srv1840490073=50} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-25 02:53:17,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-25 02:53:17,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-25 02:53:17,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-25 02:53:17,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-25 02:53:17,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-25 02:53:17,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-25 02:53:17,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-25 02:53:17,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-25 02:53:17,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-25 02:53:17,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-25 02:53:17,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-25 02:53:17,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-25 02:53:17,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-25 02:53:17,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-25 02:53:17,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-25 02:53:17,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-25 02:53:17,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-25 02:53:17,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-25 02:53:17,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-25 02:53:17,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-25 02:53:17,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-25 02:53:17,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-25 02:53:17,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-25 02:53:17,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-25 02:53:17,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-25 02:53:17,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-25 02:53:17,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-25 02:53:17,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-25 02:53:17,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-25 02:53:17,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-25 02:53:17,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-25 02:53:17,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-25 02:53:17,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-25 02:53:17,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-25 02:53:17,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-25 02:53:17,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-25 02:53:17,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-25 02:53:17,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-25 02:53:17,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-25 02:53:17,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-25 02:53:17,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-25 02:53:17,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-25 02:53:17,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-25 02:53:17,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-25 02:53:17,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-25 02:53:17,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-25 02:53:17,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-25 02:53:17,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-25 02:53:17,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-25 02:53:17,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-25 02:53:17,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-25 02:53:17,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-25 02:53:17,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-25 02:53:17,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-25 02:53:17,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-25 02:53:17,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-25 02:53:17,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-25 02:53:17,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-25 02:53:17,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-25 02:53:17,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-25 02:53:17,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-25 02:53:17,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-25 02:53:17,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-25 02:53:17,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-25 02:53:17,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-25 02:53:17,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-25 02:53:17,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-25 02:53:17,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-25 02:53:17,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-25 02:53:17,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-25 02:53:17,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-25 02:53:17,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-25 02:53:17,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-25 02:53:17,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-25 02:53:17,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-25 02:53:17,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-25 02:53:17,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-25 02:53:17,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-25 02:53:17,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-25 02:53:17,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-25 02:53:17,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-25 02:53:17,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-25 02:53:17,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-25 02:53:17,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-25 02:53:17,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-25 02:53:17,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-25 02:53:17,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-25 02:53:17,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-25 02:53:17,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-25 02:53:17,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-25 02:53:17,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-25 02:53:17,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-25 02:53:17,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-25 02:53:17,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-25 02:53:17,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-25 02:53:17,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-25 02:53:17,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-25 02:53:17,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-25 02:53:17,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-25 02:53:17,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-25 02:53:17,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-25 02:53:17,300 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04076889140689198, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06024827048572235, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04076889140689198, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17208333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23883333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-25 02:53:17,301 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-25 02:53:17,301 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17689172518894292, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04076889140689198, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06024827048572235, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04076889140689198, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17208333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23883333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-25 02:53:17,549 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 383 ms to try 5000 different iterations. Found a solution that moves 2578 regions; Going from a computed imbalance of 0.17689172518894292 to a new imbalance of 0.01594028680147536. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024285782401155285); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0412024647034992, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28644444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024285782401155285); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.046, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-25 02:53:17,559 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2762825:77 , srv1289596928:78 , srv135315108:80 , srv1831884162:80 , srv2112147671:80 , srv2136186693:80 , srv926113368:80 , srv956051400:80 , srv1513472637:81 , srv1748244579:81 , srv777123019:81 , srv1035759288:82 , srv1120220051:82 , srv1727659229:82 , srv2039496243:82 , srv453702029:82 , srv1042954044:83 , srv1229688104:83 , srv1436648944:83 , srv1595527166:83 , srv1967166138:83 , srv2016153709:83 , srv273903558:83 , srv49643950:83 , srv850337292:83 , srv1125708709:84 , srv1260853910:84 , srv140449034:84 , srv1571168493:84 , srv1625315554:84 , srv426568388:84 , srv906731712:84 , srv1124831258:85 , srv1185136671:85 , srv1297083462:85 , srv1327555290:85 , srv1358884082:85 , srv1475932371:85 , srv1513889077:85 , srv1664415141:85 , srv1750818822:85 , srv1775073491:85 , srv1932079391:85 , srv2109479112:85 , srv2115325635:85 , srv2119091690:85 , srv308944613:85 , srv332213759:85 , srv412273032:85 , srv437620312:85 , srv580544388:85 , srv761616732:85 , srv1190960103:86 , srv1424267197:86 , srv1594345136:86 , srv1617260499:86 , srv1660404233:86 , srv1800691434:86 , srv312395502:86 , srv472041493:86 , srv796412229:86 , srv824822545:86 , srv927115734:86 , srv1010718364:87 , srv1589949938:87 , srv1690311052:87 , srv1877540077:87 , srv1929793166:87 , srv1996401046:87 , srv2036544196:87 , srv206060290:87 , srv675402669:87 , srv928930748:87 , srv1039885596:88 , srv1220625257:88 , srv1227281076:88 , srv1397433796:88 , srv1474914701:88 , srv158946621:88 , srv178784677:88 , srv1957164435:88 , srv339172710:88 , srv528411091:88 , srv953282164:88 , srv1044459326:89 , srv1138712124:89 , srv153421471:89 , srv1770737853:89 , srv440071855:89 , srv865817248:89 , srv1059833356:90 , srv1264636558:90 , srv622514671:90 , srv848384040:90 , srv896165483:91 , srv1590829361:92 , srv598935079:92 , srv1840490073:93 , srv291513141:93 , srv476783668:538 } 2023-12-25 02:53:17,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-25 02:53:17,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv761616732=86, srv926113368=95, srv273903558=66, srv1327555290=19, srv2112147671=62, srv308944613=69, srv135315108=20, srv906731712=94, srv2136186693=65, srv332213759=71, srv437620312=75, srv1595527166=36, srv1664415141=40, srv1260853910=15, srv178784677=47, srv850337292=91, srv1475932371=27, srv1220625257=12, srv1831884162=49, srv896165483=93, srv1120220051=6, srv1042954044=3, srv1125708709=8, srv1967166138=55, srv1397433796=22, srv1010718364=0, srv1750818822=44, srv1932079391=53, srv1044459326=4, srv2039496243=59, srv956051400=99, srv1035759288=1, srv848384040=90, srv865817248=92, srv1800691434=48, srv2115325635=63, srv1775073491=46, srv412273032=73, srv153421471=30, srv1513889077=29, srv291513141=68, srv777123019=87, srv2762825=67, srv1594345136=35, srv1264636558=16, srv796412229=88, srv2036544196=58, srv2119091690=64, srv675402669=85, srv1660404233=39, srv1690311052=41, srv440071855=76, srv953282164=98, srv1957164435=54, srv824822545=89, srv1727659229=42, srv598935079=83, srv1289596928=17, srv580544388=82, srv1877540077=51, srv1436648944=25, srv1424267197=24, srv472041493=78, srv1190960103=11, srv1124831258=7, srv1297083462=18, srv1185136671=10, srv1625315554=38, srv339172710=72, srv2016153709=57, srv1617260499=37, srv1229688104=14, srv1748244579=43, srv1996401046=56, srv1513472637=28, srv426568388=74, srv622514671=84, srv1571168493=31, srv1590829361=34, srv2109479112=61, srv1039885596=2, srv1474914701=26, srv158946621=32, srv476783668=79, srv927115734=96, srv206060290=60, srv1227281076=13, srv528411091=81, srv1589949938=33, srv453702029=77, srv1770737853=45, srv1929793166=52, srv1059833356=5, srv49643950=80, srv1138712124=9, srv312395502=70, srv928930748=97, srv1358884082=21, srv140449034=23, srv1840490073=50} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-25 02:53:17,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-25 02:53:17,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-25 02:53:17,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-25 02:53:17,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-25 02:53:17,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-25 02:53:17,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-25 02:53:17,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-25 02:53:17,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-25 02:53:17,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-25 02:53:17,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-25 02:53:17,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-25 02:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-25 02:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-25 02:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-25 02:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-25 02:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-25 02:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-25 02:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-25 02:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-25 02:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-25 02:53:17,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-25 02:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-25 02:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-25 02:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-25 02:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-25 02:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-25 02:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-25 02:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-25 02:53:17,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-25 02:53:17,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-25 02:53:17,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-25 02:53:17,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-25 02:53:17,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-25 02:53:17,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-25 02:53:17,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-25 02:53:17,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-25 02:53:17,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-25 02:53:17,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-25 02:53:17,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-25 02:53:17,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-25 02:53:17,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-25 02:53:17,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-25 02:53:17,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-25 02:53:17,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-25 02:53:17,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-25 02:53:17,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-25 02:53:17,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-25 02:53:17,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-25 02:53:17,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-25 02:53:17,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-25 02:53:17,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-25 02:53:17,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-25 02:53:17,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-25 02:53:17,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-25 02:53:17,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-25 02:53:17,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-25 02:53:17,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-25 02:53:17,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-25 02:53:17,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-25 02:53:17,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-25 02:53:17,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-25 02:53:17,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-25 02:53:17,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-25 02:53:17,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-25 02:53:17,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-25 02:53:17,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-25 02:53:17,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-25 02:53:17,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-25 02:53:17,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-25 02:53:17,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-25 02:53:17,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-25 02:53:17,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-25 02:53:17,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-25 02:53:17,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-25 02:53:17,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-25 02:53:17,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-25 02:53:17,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-25 02:53:17,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-25 02:53:17,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-25 02:53:17,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-25 02:53:17,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-25 02:53:17,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-25 02:53:17,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-25 02:53:17,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-25 02:53:17,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-25 02:53:17,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-25 02:53:17,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-25 02:53:17,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-25 02:53:17,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-25 02:53:17,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-25 02:53:17,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-25 02:53:17,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-25 02:53:17,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-25 02:53:17,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-25 02:53:17,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-25 02:53:17,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-25 02:53:17,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-25 02:53:17,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-25 02:53:17,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-25 02:53:17,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-25 02:53:17,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-25 02:53:17,718 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024285782401155285); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0412024647034992, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024285782401155285); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.046, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-25 02:53:17,718 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-25 02:53:17,718 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015922235259652022, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024285782401155285); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0412024647034992, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024285782401155285); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.046, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-25 02:53:17,960 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 340 ms to try 5000 different iterations. Found a solution that moves 987 regions; Going from a computed imbalance of 0.015922235259652022 to a new imbalance of 0.0015539941255905746. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007439325174185004); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009063933040070526); MoveCostFunction : (multiplier=7.0, imbalance=0.10966666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007439325174185004); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.001); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.006333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-25 02:53:17,962 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2036544196:82 , srv1220625257:83 , srv153421471:83 , srv178784677:83 , srv291513141:83 , srv928930748:83 , srv1044459326:84 , srv1124831258:84 , srv1190960103:84 , srv1617260499:84 , srv1660404233:84 , srv1727659229:84 , srv1748244579:84 , srv1750818822:84 , srv312395502:84 , srv332213759:84 , srv440071855:84 , srv622514671:84 , srv953282164:84 , srv1125708709:85 , srv1264636558:85 , srv135315108:85 , srv140449034:85 , srv1424267197:85 , srv1474914701:85 , srv1475932371:85 , srv1513889077:85 , srv1589949938:85 , srv1590829361:85 , srv1594345136:85 , srv1625315554:85 , srv1775073491:85 , srv1877540077:85 , srv1932079391:85 , srv1957164435:85 , srv1996401046:85 , srv2016153709:85 , srv2039496243:85 , srv206060290:85 , srv2119091690:85 , srv308944613:85 , srv472041493:85 , srv49643950:85 , srv598935079:85 , srv761616732:85 , srv777123019:85 , srv796412229:85 , srv850337292:85 , srv896165483:85 , srv906731712:85 , srv956051400:85 , srv1010718364:86 , srv1039885596:86 , srv1042954044:86 , srv1138712124:86 , srv1227281076:86 , srv1229688104:86 , srv1260853910:86 , srv1289596928:86 , srv1327555290:86 , srv1358884082:86 , srv1513472637:86 , srv1571168493:86 , srv1664415141:86 , srv1690311052:86 , srv1840490073:86 , srv1929793166:86 , srv1967166138:86 , srv2109479112:86 , srv2112147671:86 , srv2115325635:86 , srv273903558:86 , srv2762825:86 , srv339172710:86 , srv453702029:86 , srv528411091:86 , srv675402669:86 , srv848384040:86 , srv926113368:86 , srv1059833356:87 , srv1120220051:87 , srv1297083462:87 , srv1397433796:87 , srv1436648944:87 , srv158946621:87 , srv1595527166:87 , srv1770737853:87 , srv1800691434:87 , srv1831884162:87 , srv412273032:87 , srv580544388:87 , srv824822545:87 , srv865817248:87 , srv1035759288:88 , srv1185136671:88 , srv927115734:88 , srv2136186693:89 , srv426568388:89 , srv437620312:89 , srv476783668:534 } 2023-12-25 02:53:17,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-25 02:53:17,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv761616732=86, srv926113368=95, srv273903558=66, srv1327555290=19, srv2112147671=62, srv308944613=69, srv135315108=20, srv906731712=94, srv2136186693=65, srv332213759=71, srv437620312=75, srv1595527166=36, srv1664415141=40, srv1260853910=15, srv178784677=47, srv850337292=91, srv1475932371=27, srv1220625257=12, srv1831884162=49, srv896165483=93, srv1120220051=6, srv1042954044=3, srv1125708709=8, srv1967166138=55, srv1397433796=22, srv1010718364=0, srv1750818822=44, srv1932079391=53, srv1044459326=4, srv2039496243=59, srv956051400=99, srv1035759288=1, srv848384040=90, srv865817248=92, srv1800691434=48, srv2115325635=63, srv1775073491=46, srv412273032=73, srv153421471=30, srv1513889077=29, srv291513141=68, srv777123019=87, srv2762825=67, srv1594345136=35, srv1264636558=16, srv796412229=88, srv2036544196=58, srv2119091690=64, srv675402669=85, srv1660404233=39, srv1690311052=41, srv440071855=76, srv953282164=98, srv1957164435=54, srv824822545=89, srv1727659229=42, srv598935079=83, srv1289596928=17, srv580544388=82, srv1877540077=51, srv1436648944=25, srv1424267197=24, srv472041493=78, srv1190960103=11, srv1124831258=7, srv1297083462=18, srv1185136671=10, srv1625315554=38, srv339172710=72, srv2016153709=57, srv1617260499=37, srv1229688104=14, srv1748244579=43, srv1996401046=56, srv1513472637=28, srv426568388=74, srv622514671=84, srv1571168493=31, srv1590829361=34, srv2109479112=61, srv1039885596=2, srv1474914701=26, srv158946621=32, srv476783668=79, srv927115734=96, srv206060290=60, srv1227281076=13, srv528411091=81, srv1589949938=33, srv453702029=77, srv1770737853=45, srv1929793166=52, srv1059833356=5, srv49643950=80, srv1138712124=9, srv312395502=70, srv928930748=97, srv1358884082=21, srv140449034=23, srv1840490073=50} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-25 02:53:17,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-25 02:53:17,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-25 02:53:17,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-25 02:53:17,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-25 02:53:17,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-25 02:53:17,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-25 02:53:17,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-25 02:53:17,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-25 02:53:17,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-25 02:53:17,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-25 02:53:17,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-25 02:53:17,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-25 02:53:17,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-25 02:53:17,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-25 02:53:17,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-25 02:53:17,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-25 02:53:17,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-25 02:53:17,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-25 02:53:17,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-25 02:53:17,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-25 02:53:17,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-25 02:53:17,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-25 02:53:17,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-25 02:53:17,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-25 02:53:17,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-25 02:53:17,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-25 02:53:17,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-25 02:53:17,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-25 02:53:17,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-25 02:53:17,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-25 02:53:17,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-25 02:53:17,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-25 02:53:17,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-25 02:53:17,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-25 02:53:17,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-25 02:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-25 02:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-25 02:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-25 02:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-25 02:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-25 02:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-25 02:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-25 02:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-25 02:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-25 02:53:17,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-25 02:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-25 02:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-25 02:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-25 02:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-25 02:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-25 02:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-25 02:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-25 02:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-25 02:53:17,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-25 02:53:17,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-25 02:53:17,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-25 02:53:17,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-25 02:53:17,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-25 02:53:17,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-25 02:53:17,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-25 02:53:17,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-25 02:53:17,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-25 02:53:17,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-25 02:53:17,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-25 02:53:17,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-25 02:53:17,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-25 02:53:17,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-25 02:53:17,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-25 02:53:17,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-25 02:53:17,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-25 02:53:17,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-25 02:53:17,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-25 02:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-25 02:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-25 02:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-25 02:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-25 02:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-25 02:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-25 02:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-25 02:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-25 02:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-25 02:53:17,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-25 02:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-25 02:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-25 02:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-25 02:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-25 02:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-25 02:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-25 02:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-25 02:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-25 02:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-25 02:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-25 02:53:17,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-25 02:53:17,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-25 02:53:17,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-25 02:53:17,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-25 02:53:17,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-25 02:53:17,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-25 02:53:17,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-25 02:53:17,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-25 02:53:17,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-25 02:53:18,126 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007439325174185004); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009063933040070526); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007439325174185004); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.001); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.006333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-25 02:53:18,127 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-25 02:53:18,127 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.001547083003876208, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007439325174185004); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009063933040070526); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007439325174185004); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.001); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.006333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-25 02:53:18,463 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 448 ms to try 5000 different iterations. Found a solution that moves 314 regions; Going from a computed imbalance of 0.001547083003876208 to a new imbalance of 4.3778794613511275E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.034888888888888886, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-25 02:53:18,464 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1358884082:83 , srv865817248:83 , srv1035759288:84 , srv1042954044:84 , srv1120220051:84 , srv1124831258:84 , srv1264636558:84 , srv135315108:84 , srv1397433796:84 , srv1424267197:84 , srv1436648944:84 , srv1589949938:84 , srv1595527166:84 , srv1617260499:84 , srv1748244579:84 , srv1750818822:84 , srv1770737853:84 , srv1775073491:84 , srv1800691434:84 , srv1831884162:84 , srv1840490073:84 , srv1929793166:84 , srv1932079391:84 , srv2036544196:84 , srv2039496243:84 , srv2136186693:84 , srv2762825:84 , srv308944613:84 , srv412273032:84 , srv426568388:84 , srv437620312:84 , srv440071855:84 , srv49643950:84 , srv761616732:84 , srv796412229:84 , srv824822545:84 , srv848384040:84 , srv927115734:84 , srv928930748:84 , srv953282164:84 , srv956051400:84 , srv1010718364:85 , srv1039885596:85 , srv1044459326:85 , srv1059833356:85 , srv1125708709:85 , srv1138712124:85 , srv1185136671:85 , srv1190960103:85 , srv1220625257:85 , srv1227281076:85 , srv1229688104:85 , srv1260853910:85 , srv1289596928:85 , srv1297083462:85 , srv1327555290:85 , srv1474914701:85 , srv1475932371:85 , srv1513889077:85 , srv1571168493:85 , srv158946621:85 , srv1590829361:85 , srv1594345136:85 , srv1625315554:85 , srv1660404233:85 , srv1664415141:85 , srv1690311052:85 , srv178784677:85 , srv1877540077:85 , srv1996401046:85 , srv2109479112:85 , srv2112147671:85 , srv2115325635:85 , srv2119091690:85 , srv273903558:85 , srv291513141:85 , srv312395502:85 , srv332213759:85 , srv339172710:85 , srv472041493:85 , srv528411091:85 , srv580544388:85 , srv598935079:85 , srv622514671:85 , srv675402669:85 , srv777123019:85 , srv896165483:85 , srv906731712:85 , srv926113368:85 , srv140449034:86 , srv1513472637:86 , srv153421471:86 , srv1727659229:86 , srv1957164435:86 , srv1967166138:86 , srv2016153709:86 , srv206060290:86 , srv453702029:86 , srv850337292:86 , srv476783668:618 } 2023-12-25 02:53:18,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-25 02:53:18,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv761616732=86, srv926113368=95, srv273903558=66, srv1327555290=19, srv2112147671=62, srv308944613=69, srv135315108=20, srv906731712=94, srv2136186693=65, srv332213759=71, srv437620312=75, srv1595527166=36, srv1664415141=40, srv1260853910=15, srv178784677=47, srv850337292=91, srv1475932371=27, srv1220625257=12, srv1831884162=49, srv896165483=93, srv1120220051=6, srv1042954044=3, srv1125708709=8, srv1967166138=55, srv1397433796=22, srv1010718364=0, srv1750818822=44, srv1932079391=53, srv1044459326=4, srv2039496243=59, srv956051400=99, srv1035759288=1, srv848384040=90, srv865817248=92, srv1800691434=48, srv2115325635=63, srv1775073491=46, srv412273032=73, srv153421471=30, srv1513889077=29, srv291513141=68, srv777123019=87, srv2762825=67, srv1594345136=35, srv1264636558=16, srv796412229=88, srv2036544196=58, srv2119091690=64, srv675402669=85, srv1660404233=39, srv1690311052=41, srv440071855=76, srv953282164=98, srv1957164435=54, srv824822545=89, srv1727659229=42, srv598935079=83, srv1289596928=17, srv580544388=82, srv1877540077=51, srv1436648944=25, srv1424267197=24, srv472041493=78, srv1190960103=11, srv1124831258=7, srv1297083462=18, srv1185136671=10, srv1625315554=38, srv339172710=72, srv2016153709=57, srv1617260499=37, srv1229688104=14, srv1748244579=43, srv1996401046=56, srv1513472637=28, srv426568388=74, srv622514671=84, srv1571168493=31, srv1590829361=34, srv2109479112=61, srv1039885596=2, srv1474914701=26, srv158946621=32, srv476783668=79, srv927115734=96, srv206060290=60, srv1227281076=13, srv528411091=81, srv1589949938=33, srv453702029=77, srv1770737853=45, srv1929793166=52, srv1059833356=5, srv49643950=80, srv1138712124=9, srv312395502=70, srv928930748=97, srv1358884082=21, srv140449034=23, srv1840490073=50} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-25 02:53:18,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-25 02:53:18,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-25 02:53:18,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-25 02:53:18,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-25 02:53:18,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-25 02:53:18,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-25 02:53:18,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-25 02:53:18,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-25 02:53:18,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-25 02:53:18,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-25 02:53:18,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-25 02:53:18,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-25 02:53:18,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-25 02:53:18,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-25 02:53:18,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-25 02:53:18,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-25 02:53:18,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-25 02:53:18,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-25 02:53:18,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-25 02:53:18,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-25 02:53:18,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-25 02:53:18,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-25 02:53:18,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-25 02:53:18,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-25 02:53:18,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-25 02:53:18,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-25 02:53:18,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-25 02:53:18,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-25 02:53:18,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-25 02:53:18,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-25 02:53:18,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-25 02:53:18,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-25 02:53:18,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-25 02:53:18,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-25 02:53:18,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-25 02:53:18,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-25 02:53:18,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-25 02:53:18,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-25 02:53:18,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-25 02:53:18,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-25 02:53:18,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-25 02:53:18,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-25 02:53:18,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-25 02:53:18,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-25 02:53:18,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-25 02:53:18,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-25 02:53:18,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-25 02:53:18,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-25 02:53:18,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-25 02:53:18,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-25 02:53:18,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-25 02:53:18,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-25 02:53:18,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-25 02:53:18,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-25 02:53:18,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-25 02:53:18,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-25 02:53:18,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-25 02:53:18,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-25 02:53:18,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-25 02:53:18,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-25 02:53:18,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-25 02:53:18,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-25 02:53:18,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-25 02:53:18,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-25 02:53:18,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-25 02:53:18,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-25 02:53:18,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-25 02:53:18,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-25 02:53:18,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-25 02:53:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-25 02:53:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-25 02:53:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-25 02:53:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-25 02:53:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-25 02:53:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-25 02:53:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-25 02:53:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-25 02:53:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-25 02:53:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-25 02:53:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-25 02:53:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-25 02:53:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-25 02:53:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-25 02:53:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-25 02:53:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-25 02:53:18,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-25 02:53:18,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-25 02:53:18,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-25 02:53:18,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-25 02:53:18,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-25 02:53:18,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-25 02:53:18,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-25 02:53:18,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-25 02:53:18,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-25 02:53:18,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-25 02:53:18,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-25 02:53:18,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-25 02:53:18,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-25 02:53:18,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-25 02:53:18,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-25 02:53:18,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-25 02:53:18,592 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-25 02:53:18,593 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-25 02:53:18,593 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.158011962028835E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-25 02:53:18,817 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 321 ms to try 5000 different iterations. Found a solution that moves 33 regions; Going from a computed imbalance of 4.158011962028835E-5 to a new imbalance of 7.3043102351076524E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0036666666666666666); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-25 02:53:18,817 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1227281076:83 , srv1474914701:83 , srv1513472637:83 , srv1727659229:83 , srv1929793166:83 , srv1957164435:83 , srv206060290:83 , srv312395502:83 , srv426568388:83 , srv453702029:83 , srv622514671:83 , srv1010718364:84 , srv1035759288:84 , srv1039885596:84 , srv1042954044:84 , srv1044459326:84 , srv1059833356:84 , srv1120220051:84 , srv1124831258:84 , srv1125708709:84 , srv1138712124:84 , srv1185136671:84 , srv1190960103:84 , srv1220625257:84 , srv1229688104:84 , srv1260853910:84 , srv1264636558:84 , srv1289596928:84 , srv1297083462:84 , srv1327555290:84 , srv1397433796:84 , srv140449034:84 , srv1475932371:84 , srv1513889077:84 , srv153421471:84 , srv1571168493:84 , srv158946621:84 , srv1589949938:84 , srv1590829361:84 , srv1594345136:84 , srv1595527166:84 , srv1625315554:84 , srv1660404233:84 , srv1664415141:84 , srv1690311052:84 , srv1750818822:84 , srv1775073491:84 , srv178784677:84 , srv1800691434:84 , srv1831884162:84 , srv1840490073:84 , srv1877540077:84 , srv1932079391:84 , srv1967166138:84 , srv1996401046:84 , srv2016153709:84 , srv2036544196:84 , srv2109479112:84 , srv2112147671:84 , srv2115325635:84 , srv2119091690:84 , srv2136186693:84 , srv273903558:84 , srv291513141:84 , srv332213759:84 , srv339172710:84 , srv412273032:84 , srv437620312:84 , srv440071855:84 , srv472041493:84 , srv49643950:84 , srv528411091:84 , srv580544388:84 , srv598935079:84 , srv675402669:84 , srv761616732:84 , srv777123019:84 , srv796412229:84 , srv824822545:84 , srv848384040:84 , srv850337292:84 , srv865817248:84 , srv896165483:84 , srv906731712:84 , srv926113368:84 , srv927115734:84 , srv953282164:84 , srv956051400:84 , srv135315108:85 , srv1358884082:85 , srv1424267197:85 , srv1436648944:85 , srv1617260499:85 , srv1748244579:85 , srv1770737853:85 , srv2039496243:85 , srv2762825:85 , srv308944613:85 , srv928930748:85 , srv476783668:684 } 2023-12-25 02:53:18,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-25 02:53:18,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv761616732=86, srv926113368=95, srv273903558=66, srv1327555290=19, srv2112147671=62, srv308944613=69, srv135315108=20, srv906731712=94, srv2136186693=65, srv332213759=71, srv437620312=75, srv1595527166=36, srv1664415141=40, srv1260853910=15, srv178784677=47, srv850337292=91, srv1475932371=27, srv1220625257=12, srv1831884162=49, srv896165483=93, srv1120220051=6, srv1042954044=3, srv1125708709=8, srv1967166138=55, srv1397433796=22, srv1010718364=0, srv1750818822=44, srv1932079391=53, srv1044459326=4, srv2039496243=59, srv956051400=99, srv1035759288=1, srv848384040=90, srv865817248=92, srv1800691434=48, srv2115325635=63, srv1775073491=46, srv412273032=73, srv153421471=30, srv1513889077=29, srv291513141=68, srv777123019=87, srv2762825=67, srv1594345136=35, srv1264636558=16, srv796412229=88, srv2036544196=58, srv2119091690=64, srv675402669=85, srv1660404233=39, srv1690311052=41, srv440071855=76, srv953282164=98, srv1957164435=54, srv824822545=89, srv1727659229=42, srv598935079=83, srv1289596928=17, srv580544388=82, srv1877540077=51, srv1436648944=25, srv1424267197=24, srv472041493=78, srv1190960103=11, srv1124831258=7, srv1297083462=18, srv1185136671=10, srv1625315554=38, srv339172710=72, srv2016153709=57, srv1617260499=37, srv1229688104=14, srv1748244579=43, srv1996401046=56, srv1513472637=28, srv426568388=74, srv622514671=84, srv1571168493=31, srv1590829361=34, srv2109479112=61, srv1039885596=2, srv1474914701=26, srv158946621=32, srv476783668=79, srv927115734=96, srv206060290=60, srv1227281076=13, srv528411091=81, srv1589949938=33, srv453702029=77, srv1770737853=45, srv1929793166=52, srv1059833356=5, srv49643950=80, srv1138712124=9, srv312395502=70, srv928930748=97, srv1358884082=21, srv140449034=23, srv1840490073=50} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-25 02:53:18,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-25 02:53:18,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-25 02:53:18,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-25 02:53:18,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-25 02:53:18,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-25 02:53:18,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-25 02:53:18,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-25 02:53:18,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-25 02:53:18,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-25 02:53:18,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-25 02:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-25 02:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-25 02:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-25 02:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-25 02:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-25 02:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-25 02:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-25 02:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-25 02:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-25 02:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-25 02:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-25 02:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-25 02:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-25 02:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-25 02:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-25 02:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-25 02:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-25 02:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-25 02:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-25 02:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-25 02:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-25 02:53:18,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-25 02:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-25 02:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-25 02:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-25 02:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-25 02:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-25 02:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-25 02:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-25 02:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-25 02:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-25 02:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-25 02:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-25 02:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-25 02:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-25 02:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-25 02:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-25 02:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-25 02:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-25 02:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-25 02:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-25 02:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-25 02:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-25 02:53:18,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-25 02:53:18,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-25 02:53:18,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-25 02:53:18,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-25 02:53:18,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-25 02:53:18,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-25 02:53:18,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-25 02:53:18,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-25 02:53:18,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-25 02:53:18,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-25 02:53:18,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-25 02:53:18,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-25 02:53:18,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-25 02:53:18,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-25 02:53:18,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-25 02:53:18,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-25 02:53:18,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-25 02:53:18,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-25 02:53:18,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-25 02:53:18,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-25 02:53:18,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-25 02:53:18,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-25 02:53:18,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-25 02:53:18,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-25 02:53:18,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-25 02:53:18,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-25 02:53:18,940 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-25 02:53:18,940 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1227281076:83 , srv1474914701:83 , srv1513472637:83 , srv1727659229:83 , srv1929793166:83 , srv1957164435:83 , srv206060290:83 , srv312395502:83 , srv426568388:83 , srv453702029:83 , srv622514671:83 , srv1010718364:84 , srv1035759288:84 , srv1039885596:84 , srv1042954044:84 , srv1044459326:84 , srv1059833356:84 , srv1120220051:84 , srv1124831258:84 , srv1125708709:84 , srv1138712124:84 , srv1185136671:84 , srv1190960103:84 , srv1220625257:84 , srv1229688104:84 , srv1260853910:84 , srv1264636558:84 , srv1289596928:84 , srv1297083462:84 , srv1327555290:84 , srv1397433796:84 , srv140449034:84 , srv1475932371:84 , srv1513889077:84 , srv153421471:84 , srv1571168493:84 , srv158946621:84 , srv1589949938:84 , srv1590829361:84 , srv1594345136:84 , srv1595527166:84 , srv1625315554:84 , srv1660404233:84 , srv1664415141:84 , srv1690311052:84 , srv1750818822:84 , srv1775073491:84 , srv178784677:84 , srv1800691434:84 , srv1831884162:84 , srv1840490073:84 , srv1877540077:84 , srv1932079391:84 , srv1967166138:84 , srv1996401046:84 , srv2016153709:84 , srv2036544196:84 , srv2109479112:84 , srv2112147671:84 , srv2115325635:84 , srv2119091690:84 , srv2136186693:84 , srv273903558:84 , srv291513141:84 , srv332213759:84 , srv339172710:84 , srv412273032:84 , srv437620312:84 , srv440071855:84 , srv472041493:84 , srv49643950:84 , srv528411091:84 , srv580544388:84 , srv598935079:84 , srv675402669:84 , srv761616732:84 , srv777123019:84 , srv796412229:84 , srv824822545:84 , srv848384040:84 , srv850337292:84 , srv865817248:84 , srv896165483:84 , srv906731712:84 , srv926113368:84 , srv927115734:84 , srv953282164:84 , srv956051400:84 , srv135315108:85 , srv1358884082:85 , srv1424267197:85 , srv1436648944:85 , srv1617260499:85 , srv1748244579:85 , srv1770737853:85 , srv2039496243:85 , srv2762825:85 , srv308944613:85 , srv928930748:85 , srv476783668:684 } 2023-12-25 02:53:18,963 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=284 (was 284), ProcessCount=172 (was 172), AvailableMemoryMB=11180 (was 11781) 2023-12-25 02:53:18,970 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=284, ProcessCount=172, AvailableMemoryMB=11180 2023-12-25 02:53:18,971 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-25 02:53:18,971 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-12-25 02:53:18,979 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv107774055:84 , srv1086050065:84 , srv1161751293:84 , srv1291428646:84 , srv1313928381:84 , srv1315747523:84 , srv1479821284:84 , srv1517446357:84 , srv1583161910:84 , srv1609958484:84 , srv1616946287:84 , srv1653916460:84 , srv1802729284:84 , srv1848278113:84 , srv2082899067:84 , srv2114356811:84 , srv306514508:84 , srv37906113:84 , srv401254861:84 , srv404959563:84 , srv459566202:84 , srv462386472:84 , srv524854054:84 , srv568857092:84 , srv610472624:84 , srv673533600:84 , srv745409376:84 , srv866768926:84 , srv890243294:84 , srv1437221791:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-12-25 02:53:18,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-25 02:53:18,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv524854054=23, srv745409376=27, srv462386472=22, srv610472624=25, srv1616946287=11, srv568857092=24, srv1086050065=1, srv107774055=0, srv1161751293=2, srv1609958484=10, srv2082899067=15, srv1315747523=5, srv1802729284=13, srv37906113=18, srv1313928381=4, srv2114356811=16, srv306514508=17, srv401254861=19, srv1653916460=12, srv1583161910=9, srv459566202=21, srv1437221791=6, srv866768926=28, srv1479821284=7, srv1848278113=14, srv1291428646=3, srv673533600=26, srv890243294=29, srv404959563=20, srv1517446357=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-25 02:53:18,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-25 02:53:18,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-25 02:53:18,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-25 02:53:18,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-25 02:53:18,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-25 02:53:18,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-25 02:53:18,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-25 02:53:18,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-25 02:53:18,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-25 02:53:18,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-25 02:53:18,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-25 02:53:18,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-25 02:53:18,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-25 02:53:18,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-25 02:53:18,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-25 02:53:18,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-25 02:53:18,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-25 02:53:18,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-25 02:53:18,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-25 02:53:18,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-25 02:53:18,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-25 02:53:18,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-25 02:53:18,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-25 02:53:18,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-25 02:53:18,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-25 02:53:18,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-25 02:53:18,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-25 02:53:18,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-25 02:53:18,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-25 02:53:18,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-25 02:53:18,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-25 02:53:19,004 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1136790469713194, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-25 02:53:19,005 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912551242229325, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1136790469713194, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-12-25 02:55:18,993 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6044904 different iterations. Found a solution that moves 1849 regions; Going from a computed imbalance of 0.9912551242229325 to a new imbalance of 4.3156582404131396E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6848148148148148, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-25 02:55:19,003 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv107774055:90 , srv1086050065:90 , srv1161751293:90 , srv1291428646:90 , srv1313928381:90 , srv1315747523:90 , srv1437221791:90 , srv1479821284:90 , srv1517446357:90 , srv1583161910:90 , srv1609958484:90 , srv1616946287:90 , srv1653916460:90 , srv1802729284:90 , srv1848278113:90 , srv2082899067:90 , srv2114356811:90 , srv306514508:90 , srv37906113:90 , srv401254861:90 , srv404959563:90 , srv459566202:90 , srv462386472:90 , srv524854054:90 , srv568857092:90 , srv610472624:90 , srv673533600:90 , srv745409376:90 , srv866768926:90 , srv890243294:90 } 2023-12-25 02:55:19,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-25 02:55:19,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv524854054=23, srv745409376=27, srv462386472=22, srv610472624=25, srv1616946287=11, srv568857092=24, srv1086050065=1, srv107774055=0, srv1161751293=2, srv1609958484=10, srv2082899067=15, srv1315747523=5, srv1802729284=13, srv37906113=18, srv1313928381=4, srv2114356811=16, srv306514508=17, srv401254861=19, srv1653916460=12, srv1583161910=9, srv459566202=21, srv1437221791=6, srv866768926=28, srv1479821284=7, srv1848278113=14, srv1291428646=3, srv673533600=26, srv890243294=29, srv404959563=20, srv1517446357=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-25 02:55:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-25 02:55:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-25 02:55:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-25 02:55:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-25 02:55:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-25 02:55:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-25 02:55:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-25 02:55:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-25 02:55:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-25 02:55:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-25 02:55:19,013 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-25 02:55:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-25 02:55:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-25 02:55:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-25 02:55:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-25 02:55:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-25 02:55:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-25 02:55:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-25 02:55:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-25 02:55:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-25 02:55:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-25 02:55:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-25 02:55:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-25 02:55:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-25 02:55:19,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-25 02:55:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-25 02:55:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-25 02:55:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-25 02:55:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-25 02:55:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-25 02:55:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-25 02:55:19,033 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-25 02:55:19,034 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv107774055:90 , srv1086050065:90 , srv1161751293:90 , srv1291428646:90 , srv1313928381:90 , srv1315747523:90 , srv1437221791:90 , srv1479821284:90 , srv1517446357:90 , srv1583161910:90 , srv1609958484:90 , srv1616946287:90 , srv1653916460:90 , srv1802729284:90 , srv1848278113:90 , srv2082899067:90 , srv2114356811:90 , srv306514508:90 , srv37906113:90 , srv401254861:90 , srv404959563:90 , srv459566202:90 , srv462386472:90 , srv524854054:90 , srv568857092:90 , srv610472624:90 , srv673533600:90 , srv745409376:90 , srv866768926:90 , srv890243294:90 } 2023-12-25 02:55:19,054 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=229 (was 284), ProcessCount=168 (was 172), AvailableMemoryMB=10823 (was 11180)