2023-12-24 23:53:21,094 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-12-24 23:53:21,429 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-12-24 23:53:21,549 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-12-24 23:53:21,620 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-24 23:53:21,627 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-24 23:53:21,643 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=291, ProcessCount=169, AvailableMemoryMB=11751 2023-12-24 23:53:21,646 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-24 23:53:21,646 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-12-24 23:53:21,787 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv100905562:84 , srv1039848010:84 , srv1068851679:84 , srv1086051252:84 , srv1106275516:84 , srv1116029968:84 , srv1124773312:84 , srv1164252757:84 , srv1171409751:84 , srv1178138356:84 , srv1179544135:84 , srv1204619281:84 , srv1222202866:84 , srv1229603231:84 , srv1271043261:84 , srv1296721447:84 , srv1304850798:84 , srv1310751611:84 , srv135363050:84 , srv1412625480:84 , srv1463924446:84 , srv1464311664:84 , srv1474514663:84 , srv1476290207:84 , srv1479846093:84 , srv1498813924:84 , srv1499536110:84 , srv1523795877:84 , srv1540873064:84 , srv1544707181:84 , srv155288084:84 , srv1569961506:84 , srv1583134907:84 , srv158584979:84 , srv1599324:84 , srv1609135966:84 , srv1611227535:84 , srv1613063494:84 , srv1625481389:84 , srv1664413401:84 , srv1755361266:84 , srv1758125369:84 , srv1828014968:84 , srv1831225009:84 , srv1888934480:84 , srv1919448122:84 , srv1976134831:84 , srv1996390770:84 , srv1998558817:84 , srv2004416148:84 , srv2008955676:84 , srv2021037653:84 , srv2031787984:84 , srv2080530134:84 , srv2113018930:84 , srv2116161037:84 , srv2134816287:84 , srv275235570:84 , srv290248788:84 , srv331095736:84 , srv374136235:84 , srv407903732:84 , srv408936987:84 , srv415204158:84 , srv416020971:84 , srv416245105:84 , srv423682277:84 , srv452114297:84 , srv477121615:84 , srv493086728:84 , srv495896058:84 , srv509018606:84 , srv515594154:84 , srv527201044:84 , srv529054392:84 , srv574573162:84 , srv576948304:84 , srv60237876:84 , srv606011854:84 , srv613169750:84 , srv613271476:84 , srv647946399:84 , srv674802528:84 , srv68642599:84 , srv686443752:84 , srv699754202:84 , srv712594253:84 , srv732418113:84 , srv742629624:84 , srv767258427:84 , srv812437322:84 , srv827287294:84 , srv877317608:84 , srv95226212:84 , srv953263816:84 , srv954711736:84 , srv968642543:84 , srv977877422:84 , srv984709116:84 , srv959125770:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-12-24 23:53:21,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-24 23:53:21,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1540873064=28, srv1755361266=40, srv290248788=58, srv613169750=79, srv1523795877=27, srv1271043261=14, srv1664413401=39, srv423682277=66, srv1171409751=8, srv742629624=88, srv477121615=68, srv1569961506=31, srv1599324=34, srv977877422=98, srv1976134831=46, srv1463924446=20, srv1499536110=26, srv1310751611=17, srv1412625480=19, srv574573162=75, srv407903732=61, srv699754202=85, srv712594253=86, srv1544707181=29, srv416245105=65, srv2113018930=54, srv1124773312=6, srv331095736=59, srv576948304=76, srv606011854=78, srv827287294=91, srv1611227535=36, srv529054392=74, srv452114297=67, srv953263816=94, srv1116029968=5, srv1474514663=22, srv408936987=62, srv415204158=63, srv68642599=83, srv2116161037=55, srv495896058=70, srv1068851679=2, srv686443752=84, srv959125770=96, srv2080530134=53, srv2134816287=56, srv1479846093=24, srv1613063494=37, srv1222202866=12, srv275235570=57, srv60237876=77, srv155288084=30, srv968642543=97, srv2031787984=52, srv732418113=87, srv1476290207=23, srv613271476=80, srv135363050=18, srv1106275516=4, srv674802528=82, srv877317608=92, srv1039848010=1, srv767258427=89, srv954711736=95, srv1998558817=48, srv812437322=90, srv374136235=60, srv2008955676=50, srv509018606=71, srv1178138356=9, srv1583134907=32, srv1919448122=45, srv1888934480=44, srv1296721447=15, srv95226212=93, srv647946399=81, srv1609135966=35, srv1204619281=11, srv1758125369=41, srv1086051252=3, srv1828014968=42, srv2021037653=51, srv515594154=72, srv1164252757=7, srv1304850798=16, srv1831225009=43, srv1996390770=47, srv1179544135=10, srv2004416148=49, srv416020971=64, srv527201044=73, srv1498813924=25, srv1625481389=38, srv984709116=99, srv100905562=0, srv1464311664=21, srv158584979=33, srv1229603231=13, srv493086728=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-24 23:53:21,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-24 23:53:21,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-24 23:53:21,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-24 23:53:21,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-24 23:53:21,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-24 23:53:21,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-24 23:53:21,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-24 23:53:21,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-24 23:53:21,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-24 23:53:21,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-24 23:53:21,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-24 23:53:21,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-24 23:53:21,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-24 23:53:21,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-24 23:53:21,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-24 23:53:21,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-24 23:53:21,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-24 23:53:21,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-24 23:53:21,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-24 23:53:21,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-24 23:53:21,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-24 23:53:21,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-24 23:53:21,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-24 23:53:21,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-24 23:53:21,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-24 23:53:21,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-24 23:53:21,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-24 23:53:21,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-24 23:53:21,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-24 23:53:21,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-24 23:53:21,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-24 23:53:21,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-24 23:53:21,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-24 23:53:21,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-24 23:53:21,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-24 23:53:21,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-24 23:53:21,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-24 23:53:21,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-24 23:53:21,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-24 23:53:21,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-24 23:53:21,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-24 23:53:21,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-24 23:53:21,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-24 23:53:21,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-24 23:53:21,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-24 23:53:21,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-24 23:53:21,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-24 23:53:21,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-24 23:53:21,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-24 23:53:21,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-24 23:53:21,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-24 23:53:21,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-24 23:53:21,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-24 23:53:21,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-24 23:53:21,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-24 23:53:21,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-24 23:53:21,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-24 23:53:21,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-24 23:53:21,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-24 23:53:21,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-24 23:53:21,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-24 23:53:21,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-24 23:53:21,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-24 23:53:21,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-24 23:53:21,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-24 23:53:21,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-24 23:53:21,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-24 23:53:21,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-24 23:53:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-24 23:53:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-24 23:53:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-24 23:53:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-24 23:53:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-24 23:53:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-24 23:53:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-24 23:53:21,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-24 23:53:21,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-24 23:53:21,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-24 23:53:21,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-24 23:53:21,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-24 23:53:21,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-24 23:53:21,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-24 23:53:21,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-24 23:53:21,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-24 23:53:21,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-24 23:53:21,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-24 23:53:21,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-24 23:53:21,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-24 23:53:21,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-24 23:53:21,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-24 23:53:21,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-24 23:53:21,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-24 23:53:21,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-24 23:53:21,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-24 23:53:21,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-24 23:53:21,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-24 23:53:21,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-24 23:53:21,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-24 23:53:21,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-24 23:53:21,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-24 23:53:21,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-24 23:53:22,127 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-24 23:53:22,128 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-24 23:53:22,128 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-24 23:53:22,486 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 518 ms to try 5000 different iterations. Found a solution that moves 4674 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18357505307868857. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04006634975999387, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05345734739978523, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5193333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04006634975999387, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.179, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24391666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-24 23:53:22,513 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1086051252:81 , srv495896058:81 , srv674802528:81 , srv2004416148:82 , srv408936987:82 , srv1039848010:83 , srv1124773312:83 , srv1222202866:83 , srv1304850798:83 , srv155288084:83 , srv158584979:83 , srv1611227535:83 , srv2134816287:83 , srv416245105:83 , srv827287294:83 , srv1463924446:84 , srv1476290207:84 , srv1831225009:84 , srv1996390770:84 , srv2080530134:84 , srv606011854:84 , srv613169750:84 , srv742629624:84 , srv767258427:84 , srv877317608:84 , srv1229603231:85 , srv135363050:85 , srv1464311664:85 , srv1474514663:85 , srv1583134907:85 , srv1609135966:85 , srv1613063494:85 , srv1828014968:85 , srv1976134831:85 , srv407903732:85 , srv415204158:85 , srv452114297:85 , srv515594154:85 , srv574573162:85 , srv977877422:85 , srv1068851679:86 , srv1498813924:86 , srv1523795877:86 , srv1544707181:86 , srv1625481389:86 , srv2116161037:86 , srv331095736:86 , srv374136235:86 , srv60237876:86 , srv647946399:86 , srv732418113:86 , srv95226212:86 , srv954711736:86 , srv968642543:86 , srv1178138356:87 , srv1179544135:87 , srv1271043261:87 , srv1540873064:87 , srv1664413401:87 , srv2021037653:87 , srv275235570:87 , srv290248788:87 , srv423682277:87 , srv493086728:87 , srv984709116:87 , srv100905562:88 , srv1116029968:88 , srv1164252757:88 , srv1171409751:88 , srv1204619281:88 , srv1569961506:88 , srv1599324:88 , srv1998558817:88 , srv2008955676:88 , srv509018606:88 , srv699754202:88 , srv812437322:88 , srv1106275516:89 , srv1479846093:89 , srv1758125369:89 , srv1888934480:89 , srv1919448122:89 , srv686443752:89 , srv712594253:89 , srv1412625480:90 , srv1499536110:90 , srv2031787984:90 , srv2113018930:90 , srv527201044:90 , srv529054392:90 , srv953263816:90 , srv1296721447:91 , srv1310751611:91 , srv1755361266:91 , srv68642599:91 , srv416020971:92 , srv477121615:92 , srv576948304:92 , srv613271476:92 , srv959125770:446 } 2023-12-24 23:53:22,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-24 23:53:22,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1540873064=28, srv1755361266=40, srv290248788=58, srv613169750=79, srv1523795877=27, srv1271043261=14, srv1664413401=39, srv423682277=66, srv1171409751=8, srv742629624=88, srv477121615=68, srv1569961506=31, srv1599324=34, srv977877422=98, srv1976134831=46, srv1463924446=20, srv1499536110=26, srv1310751611=17, srv1412625480=19, srv574573162=75, srv407903732=61, srv699754202=85, srv712594253=86, srv1544707181=29, srv416245105=65, srv2113018930=54, srv1124773312=6, srv331095736=59, srv576948304=76, srv606011854=78, srv827287294=91, srv1611227535=36, srv529054392=74, srv452114297=67, srv953263816=94, srv1116029968=5, srv1474514663=22, srv408936987=62, srv415204158=63, srv68642599=83, srv2116161037=55, srv495896058=70, srv1068851679=2, srv686443752=84, srv959125770=96, srv2080530134=53, srv2134816287=56, srv1479846093=24, srv1613063494=37, srv1222202866=12, srv275235570=57, srv60237876=77, srv155288084=30, srv968642543=97, srv2031787984=52, srv732418113=87, srv1476290207=23, srv613271476=80, srv135363050=18, srv1106275516=4, srv674802528=82, srv877317608=92, srv1039848010=1, srv767258427=89, srv954711736=95, srv1998558817=48, srv812437322=90, srv374136235=60, srv2008955676=50, srv509018606=71, srv1178138356=9, srv1583134907=32, srv1919448122=45, srv1888934480=44, srv1296721447=15, srv95226212=93, srv647946399=81, srv1609135966=35, srv1204619281=11, srv1758125369=41, srv1086051252=3, srv1828014968=42, srv2021037653=51, srv515594154=72, srv1164252757=7, srv1304850798=16, srv1831225009=43, srv1996390770=47, srv1179544135=10, srv2004416148=49, srv416020971=64, srv527201044=73, srv1498813924=25, srv1625481389=38, srv984709116=99, srv100905562=0, srv1464311664=21, srv158584979=33, srv1229603231=13, srv493086728=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-24 23:53:22,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-24 23:53:22,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-24 23:53:22,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-24 23:53:22,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-24 23:53:22,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-24 23:53:22,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-24 23:53:22,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-24 23:53:22,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-24 23:53:22,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-24 23:53:22,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-24 23:53:22,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-24 23:53:22,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-24 23:53:22,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-24 23:53:22,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-24 23:53:22,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-24 23:53:22,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-24 23:53:22,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-24 23:53:22,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-24 23:53:22,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-24 23:53:22,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-24 23:53:22,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-24 23:53:22,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-24 23:53:22,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-24 23:53:22,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-24 23:53:22,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-24 23:53:22,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-24 23:53:22,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-24 23:53:22,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-24 23:53:22,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-24 23:53:22,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-24 23:53:22,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-24 23:53:22,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-24 23:53:22,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-24 23:53:22,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-24 23:53:22,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-24 23:53:22,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-24 23:53:22,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-24 23:53:22,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-24 23:53:22,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-24 23:53:22,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-24 23:53:22,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-24 23:53:22,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-24 23:53:22,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-24 23:53:22,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-24 23:53:22,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-24 23:53:22,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-24 23:53:22,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-24 23:53:22,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-24 23:53:22,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-24 23:53:22,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-24 23:53:22,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-24 23:53:22,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-24 23:53:22,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-24 23:53:22,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-24 23:53:22,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-24 23:53:22,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-24 23:53:22,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-24 23:53:22,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-24 23:53:22,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-24 23:53:22,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-24 23:53:22,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-24 23:53:22,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-24 23:53:22,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-24 23:53:22,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-24 23:53:22,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-24 23:53:22,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-24 23:53:22,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-24 23:53:22,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-24 23:53:22,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-24 23:53:22,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-24 23:53:22,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-24 23:53:22,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-24 23:53:22,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-24 23:53:22,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-24 23:53:22,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-24 23:53:22,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-24 23:53:22,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-24 23:53:22,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-24 23:53:22,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-24 23:53:22,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-24 23:53:22,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-24 23:53:22,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-24 23:53:22,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-24 23:53:22,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-24 23:53:22,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-24 23:53:22,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-24 23:53:22,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-24 23:53:22,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-24 23:53:22,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-24 23:53:22,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-24 23:53:22,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-24 23:53:22,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-24 23:53:22,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-24 23:53:22,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-24 23:53:22,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-24 23:53:22,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-24 23:53:22,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-24 23:53:22,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-24 23:53:22,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-24 23:53:22,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-24 23:53:22,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-24 23:53:22,713 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04006634975999387, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05345734739978523, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04006634975999387, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.179, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24391666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-24 23:53:22,713 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-24 23:53:22,713 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18354232503117798, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04006634975999387, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05345734739978523, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04006634975999387, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.179, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24391666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-24 23:53:22,963 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 378 ms to try 5000 different iterations. Found a solution that moves 2670 regions; Going from a computed imbalance of 0.18354232503117798 to a new imbalance of 0.01421454765520778. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020694392658813322); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029855619650098673, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2966666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020694392658813322); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04175, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-24 23:53:22,974 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv576948304:79 , srv1755361266:80 , srv613271476:80 , srv68642599:80 , srv1296721447:81 , srv1499536110:81 , srv416020971:81 , srv1310751611:82 , srv1479846093:82 , srv1888934480:82 , srv2031787984:82 , srv477121615:82 , srv1106275516:83 , srv1164252757:83 , srv1412625480:83 , srv1569961506:83 , srv1758125369:83 , srv2008955676:83 , srv2113018930:83 , srv423682277:83 , srv527201044:83 , srv529054392:83 , srv699754202:83 , srv953263816:83 , srv100905562:84 , srv1271043261:84 , srv1625481389:84 , srv1919448122:84 , srv493086728:84 , srv686443752:84 , srv1124773312:85 , srv1171409751:85 , srv1178138356:85 , srv1204619281:85 , srv135363050:85 , srv1540873064:85 , srv1599324:85 , srv1664413401:85 , srv290248788:85 , srv509018606:85 , srv712594253:85 , srv984709116:85 , srv1116029968:86 , srv1179544135:86 , srv1229603231:86 , srv1474514663:86 , srv1476290207:86 , srv1523795877:86 , srv1544707181:86 , srv1609135966:86 , srv1976134831:86 , srv1996390770:86 , srv2021037653:86 , srv275235570:86 , srv374136235:86 , srv574573162:86 , srv60237876:86 , srv647946399:86 , srv732418113:86 , srv812437322:86 , srv95226212:86 , srv954711736:86 , srv1464311664:87 , srv1498813924:87 , srv1828014968:87 , srv1998558817:87 , srv2116161037:87 , srv407903732:87 , srv1039848010:88 , srv1068851679:88 , srv1222202866:88 , srv1463924446:88 , srv1583134907:88 , srv1613063494:88 , srv1831225009:88 , srv331095736:88 , srv408936987:88 , srv415204158:88 , srv515594154:88 , srv613169750:88 , srv742629624:88 , srv767258427:88 , srv827287294:88 , srv877317608:88 , srv977877422:88 , srv1304850798:89 , srv158584979:89 , srv1611227535:89 , srv2080530134:89 , srv2134816287:89 , srv452114297:89 , srv606011854:89 , srv968642543:89 , srv416245105:90 , srv674802528:90 , srv1086051252:91 , srv155288084:91 , srv495896058:91 , srv2004416148:93 , srv959125770:512 } 2023-12-24 23:53:22,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-24 23:53:22,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1540873064=28, srv1755361266=40, srv290248788=58, srv613169750=79, srv1523795877=27, srv1271043261=14, srv1664413401=39, srv423682277=66, srv1171409751=8, srv742629624=88, srv477121615=68, srv1569961506=31, srv1599324=34, srv977877422=98, srv1976134831=46, srv1463924446=20, srv1499536110=26, srv1310751611=17, srv1412625480=19, srv574573162=75, srv407903732=61, srv699754202=85, srv712594253=86, srv1544707181=29, srv416245105=65, srv2113018930=54, srv1124773312=6, srv331095736=59, srv576948304=76, srv606011854=78, srv827287294=91, srv1611227535=36, srv529054392=74, srv452114297=67, srv953263816=94, srv1116029968=5, srv1474514663=22, srv408936987=62, srv415204158=63, srv68642599=83, srv2116161037=55, srv495896058=70, srv1068851679=2, srv686443752=84, srv959125770=96, srv2080530134=53, srv2134816287=56, srv1479846093=24, srv1613063494=37, srv1222202866=12, srv275235570=57, srv60237876=77, srv155288084=30, srv968642543=97, srv2031787984=52, srv732418113=87, srv1476290207=23, srv613271476=80, srv135363050=18, srv1106275516=4, srv674802528=82, srv877317608=92, srv1039848010=1, srv767258427=89, srv954711736=95, srv1998558817=48, srv812437322=90, srv374136235=60, srv2008955676=50, srv509018606=71, srv1178138356=9, srv1583134907=32, srv1919448122=45, srv1888934480=44, srv1296721447=15, srv95226212=93, srv647946399=81, srv1609135966=35, srv1204619281=11, srv1758125369=41, srv1086051252=3, srv1828014968=42, srv2021037653=51, srv515594154=72, srv1164252757=7, srv1304850798=16, srv1831225009=43, srv1996390770=47, srv1179544135=10, srv2004416148=49, srv416020971=64, srv527201044=73, srv1498813924=25, srv1625481389=38, srv984709116=99, srv100905562=0, srv1464311664=21, srv158584979=33, srv1229603231=13, srv493086728=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-24 23:53:23,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-24 23:53:23,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-24 23:53:23,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-24 23:53:23,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-24 23:53:23,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-24 23:53:23,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-24 23:53:23,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-24 23:53:23,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-24 23:53:23,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-24 23:53:23,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-24 23:53:23,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-24 23:53:23,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-24 23:53:23,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-24 23:53:23,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-24 23:53:23,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-24 23:53:23,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-24 23:53:23,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-24 23:53:23,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-24 23:53:23,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-24 23:53:23,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-24 23:53:23,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-24 23:53:23,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-24 23:53:23,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-24 23:53:23,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-24 23:53:23,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-24 23:53:23,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-24 23:53:23,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-24 23:53:23,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-24 23:53:23,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-24 23:53:23,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-24 23:53:23,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-24 23:53:23,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-24 23:53:23,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-24 23:53:23,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-24 23:53:23,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-24 23:53:23,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-24 23:53:23,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-24 23:53:23,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-24 23:53:23,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-24 23:53:23,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-24 23:53:23,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-24 23:53:23,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-24 23:53:23,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-24 23:53:23,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-24 23:53:23,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-24 23:53:23,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-24 23:53:23,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-24 23:53:23,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-24 23:53:23,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-24 23:53:23,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-24 23:53:23,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-24 23:53:23,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-24 23:53:23,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-24 23:53:23,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-24 23:53:23,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-24 23:53:23,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-24 23:53:23,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-24 23:53:23,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-24 23:53:23,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-24 23:53:23,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-24 23:53:23,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-24 23:53:23,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-24 23:53:23,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-24 23:53:23,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-24 23:53:23,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-24 23:53:23,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-24 23:53:23,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-24 23:53:23,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-24 23:53:23,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-24 23:53:23,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-24 23:53:23,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-24 23:53:23,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-24 23:53:23,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-24 23:53:23,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-24 23:53:23,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-24 23:53:23,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-24 23:53:23,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-24 23:53:23,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-24 23:53:23,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-24 23:53:23,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-24 23:53:23,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-24 23:53:23,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-24 23:53:23,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-24 23:53:23,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-24 23:53:23,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-24 23:53:23,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-24 23:53:23,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-24 23:53:23,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-24 23:53:23,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-24 23:53:23,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-24 23:53:23,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-24 23:53:23,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-24 23:53:23,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-24 23:53:23,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-24 23:53:23,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-24 23:53:23,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-24 23:53:23,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-24 23:53:23,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-24 23:53:23,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-24 23:53:23,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-24 23:53:23,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-24 23:53:23,130 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020694392658813322); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029855619650098673, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020694392658813322); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04175, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-24 23:53:23,130 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-24 23:53:23,130 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014195851915615725, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020694392658813322); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029855619650098673, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020694392658813322); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04175, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-24 23:53:23,393 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 362 ms to try 5000 different iterations. Found a solution that moves 988 regions; Going from a computed imbalance of 0.014195851915615725 to a new imbalance of 6.212832619059024E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005327531449513396); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00578321422557561); MoveCostFunction : (multiplier=7.0, imbalance=0.10977777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005327531449513396); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.002916666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-24 23:53:23,395 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1116029968:82 , srv2004416148:82 , srv1068851679:83 , srv155288084:83 , srv1998558817:83 , srv812437322:83 , srv968642543:83 , srv1204619281:84 , srv1310751611:84 , srv1412625480:84 , srv1583134907:84 , srv1599324:84 , srv1613063494:84 , srv2113018930:84 , srv275235570:84 , srv331095736:84 , srv416020971:84 , srv452114297:84 , srv477121615:84 , srv509018606:84 , srv515594154:84 , srv527201044:84 , srv606011854:84 , srv712594253:84 , srv953263816:84 , srv977877422:84 , srv1086051252:85 , srv1171409751:85 , srv1179544135:85 , srv1463924446:85 , srv1498813924:85 , srv1523795877:85 , srv1540873064:85 , srv1544707181:85 , srv1758125369:85 , srv1828014968:85 , srv1919448122:85 , srv2021037653:85 , srv2134816287:85 , srv407903732:85 , srv415204158:85 , srv529054392:85 , srv60237876:85 , srv613169750:85 , srv613271476:85 , srv647946399:85 , srv732418113:85 , srv767258427:85 , srv100905562:86 , srv1039848010:86 , srv1106275516:86 , srv1164252757:86 , srv1229603231:86 , srv1296721447:86 , srv1304850798:86 , srv1464311664:86 , srv1499536110:86 , srv1569961506:86 , srv158584979:86 , srv1609135966:86 , srv1611227535:86 , srv1664413401:86 , srv1755361266:86 , srv1831225009:86 , srv1888934480:86 , srv1976134831:86 , srv2008955676:86 , srv2031787984:86 , srv2080530134:86 , srv290248788:86 , srv374136235:86 , srv416245105:86 , srv495896058:86 , srv574573162:86 , srv576948304:86 , srv674802528:86 , srv68642599:86 , srv686443752:86 , srv699754202:86 , srv742629624:86 , srv827287294:86 , srv877317608:86 , srv95226212:86 , srv984709116:86 , srv1178138356:87 , srv1222202866:87 , srv1271043261:87 , srv1479846093:87 , srv1625481389:87 , srv1996390770:87 , srv2116161037:87 , srv423682277:87 , srv493086728:87 , srv954711736:87 , srv135363050:88 , srv1474514663:88 , srv1476290207:88 , srv408936987:88 , srv1124773312:90 , srv959125770:547 } 2023-12-24 23:53:23,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-24 23:53:23,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1540873064=28, srv1755361266=40, srv290248788=58, srv613169750=79, srv1523795877=27, srv1271043261=14, srv1664413401=39, srv423682277=66, srv1171409751=8, srv742629624=88, srv477121615=68, srv1569961506=31, srv1599324=34, srv977877422=98, srv1976134831=46, srv1463924446=20, srv1499536110=26, srv1310751611=17, srv1412625480=19, srv574573162=75, srv407903732=61, srv699754202=85, srv712594253=86, srv1544707181=29, srv416245105=65, srv2113018930=54, srv1124773312=6, srv331095736=59, srv576948304=76, srv606011854=78, srv827287294=91, srv1611227535=36, srv529054392=74, srv452114297=67, srv953263816=94, srv1116029968=5, srv1474514663=22, srv408936987=62, srv415204158=63, srv68642599=83, srv2116161037=55, srv495896058=70, srv1068851679=2, srv686443752=84, srv959125770=96, srv2080530134=53, srv2134816287=56, srv1479846093=24, srv1613063494=37, srv1222202866=12, srv275235570=57, srv60237876=77, srv155288084=30, srv968642543=97, srv2031787984=52, srv732418113=87, srv1476290207=23, srv613271476=80, srv135363050=18, srv1106275516=4, srv674802528=82, srv877317608=92, srv1039848010=1, srv767258427=89, srv954711736=95, srv1998558817=48, srv812437322=90, srv374136235=60, srv2008955676=50, srv509018606=71, srv1178138356=9, srv1583134907=32, srv1919448122=45, srv1888934480=44, srv1296721447=15, srv95226212=93, srv647946399=81, srv1609135966=35, srv1204619281=11, srv1758125369=41, srv1086051252=3, srv1828014968=42, srv2021037653=51, srv515594154=72, srv1164252757=7, srv1304850798=16, srv1831225009=43, srv1996390770=47, srv1179544135=10, srv2004416148=49, srv416020971=64, srv527201044=73, srv1498813924=25, srv1625481389=38, srv984709116=99, srv100905562=0, srv1464311664=21, srv158584979=33, srv1229603231=13, srv493086728=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-24 23:53:23,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-24 23:53:23,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-24 23:53:23,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-24 23:53:23,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-24 23:53:23,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-24 23:53:23,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-24 23:53:23,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-24 23:53:23,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-24 23:53:23,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-24 23:53:23,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-24 23:53:23,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-24 23:53:23,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-24 23:53:23,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-24 23:53:23,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-24 23:53:23,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-24 23:53:23,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-24 23:53:23,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-24 23:53:23,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-24 23:53:23,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-24 23:53:23,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-24 23:53:23,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-24 23:53:23,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-24 23:53:23,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-24 23:53:23,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-24 23:53:23,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-24 23:53:23,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-24 23:53:23,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-24 23:53:23,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-24 23:53:23,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-24 23:53:23,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-24 23:53:23,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-24 23:53:23,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-24 23:53:23,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-24 23:53:23,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-24 23:53:23,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-24 23:53:23,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-24 23:53:23,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-24 23:53:23,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-24 23:53:23,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-24 23:53:23,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-24 23:53:23,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-24 23:53:23,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-24 23:53:23,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-24 23:53:23,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-24 23:53:23,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-24 23:53:23,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-24 23:53:23,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-24 23:53:23,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-24 23:53:23,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-24 23:53:23,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-24 23:53:23,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-24 23:53:23,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-24 23:53:23,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-24 23:53:23,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-24 23:53:23,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-24 23:53:23,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-24 23:53:23,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-24 23:53:23,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-24 23:53:23,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-24 23:53:23,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-24 23:53:23,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-24 23:53:23,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-24 23:53:23,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-24 23:53:23,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-24 23:53:23,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-24 23:53:23,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-24 23:53:23,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-24 23:53:23,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-24 23:53:23,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-24 23:53:23,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-24 23:53:23,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-24 23:53:23,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-24 23:53:23,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-24 23:53:23,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-24 23:53:23,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-24 23:53:23,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-24 23:53:23,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-24 23:53:23,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-24 23:53:23,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-24 23:53:23,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-24 23:53:23,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-24 23:53:23,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-24 23:53:23,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-24 23:53:23,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-24 23:53:23,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-24 23:53:23,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-24 23:53:23,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-24 23:53:23,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-24 23:53:23,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-24 23:53:23,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-24 23:53:23,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-24 23:53:23,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-24 23:53:23,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-24 23:53:23,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-24 23:53:23,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-24 23:53:23,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-24 23:53:23,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-24 23:53:23,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-24 23:53:23,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-24 23:53:23,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-24 23:53:23,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-24 23:53:23,534 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005327531449513396); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00578321422557561); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005327531449513396); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.002916666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-24 23:53:23,534 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-12-24 23:53:23,535 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.143651380418761E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005327531449513396); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00578321422557561); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005327531449513396); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.002916666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-12-24 23:53:23,905 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 474 ms to try 5000 different iterations. Found a solution that moves 204 regions; Going from a computed imbalance of 6.143651380418761E-4 to a new imbalance of 1.801936302069529E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.02266666666666667); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-24 23:53:23,906 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2116161037:82 , srv1178138356:83 , srv1474514663:83 , srv1479846093:83 , srv2080530134:83 , srv416245105:83 , srv686443752:83 , srv742629624:83 , srv954711736:83 , srv100905562:84 , srv1086051252:84 , srv1106275516:84 , srv1124773312:84 , srv1171409751:84 , srv1179544135:84 , srv1222202866:84 , srv1271043261:84 , srv1296721447:84 , srv1304850798:84 , srv135363050:84 , srv1464311664:84 , srv1476290207:84 , srv1498813924:84 , srv1540873064:84 , srv1611227535:84 , srv1664413401:84 , srv1831225009:84 , srv1919448122:84 , srv1998558817:84 , srv2021037653:84 , srv2031787984:84 , srv290248788:84 , srv331095736:84 , srv374136235:84 , srv408936987:84 , srv415204158:84 , srv452114297:84 , srv477121615:84 , srv493086728:84 , srv495896058:84 , srv529054392:84 , srv576948304:84 , srv712594253:84 , srv877317608:84 , srv95226212:84 , srv968642543:84 , srv1039848010:85 , srv1068851679:85 , srv1164252757:85 , srv1229603231:85 , srv1310751611:85 , srv1412625480:85 , srv1463924446:85 , srv1499536110:85 , srv1523795877:85 , srv1544707181:85 , srv155288084:85 , srv1569961506:85 , srv1583134907:85 , srv158584979:85 , srv1599324:85 , srv1609135966:85 , srv1613063494:85 , srv1625481389:85 , srv1755361266:85 , srv1758125369:85 , srv1828014968:85 , srv1888934480:85 , srv1996390770:85 , srv2004416148:85 , srv2008955676:85 , srv2113018930:85 , srv2134816287:85 , srv275235570:85 , srv407903732:85 , srv416020971:85 , srv423682277:85 , srv509018606:85 , srv515594154:85 , srv527201044:85 , srv574573162:85 , srv60237876:85 , srv606011854:85 , srv613169750:85 , srv613271476:85 , srv647946399:85 , srv674802528:85 , srv68642599:85 , srv699754202:85 , srv732418113:85 , srv767258427:85 , srv827287294:85 , srv953263816:85 , srv977877422:85 , srv984709116:85 , srv1116029968:86 , srv1204619281:86 , srv1976134831:86 , srv812437322:86 , srv959125770:637 } 2023-12-24 23:53:23,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-24 23:53:23,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1540873064=28, srv1755361266=40, srv290248788=58, srv613169750=79, srv1523795877=27, srv1271043261=14, srv1664413401=39, srv423682277=66, srv1171409751=8, srv742629624=88, srv477121615=68, srv1569961506=31, srv1599324=34, srv977877422=98, srv1976134831=46, srv1463924446=20, srv1499536110=26, srv1310751611=17, srv1412625480=19, srv574573162=75, srv407903732=61, srv699754202=85, srv712594253=86, srv1544707181=29, srv416245105=65, srv2113018930=54, srv1124773312=6, srv331095736=59, srv576948304=76, srv606011854=78, srv827287294=91, srv1611227535=36, srv529054392=74, srv452114297=67, srv953263816=94, srv1116029968=5, srv1474514663=22, srv408936987=62, srv415204158=63, srv68642599=83, srv2116161037=55, srv495896058=70, srv1068851679=2, srv686443752=84, srv959125770=96, srv2080530134=53, srv2134816287=56, srv1479846093=24, srv1613063494=37, srv1222202866=12, srv275235570=57, srv60237876=77, srv155288084=30, srv968642543=97, srv2031787984=52, srv732418113=87, srv1476290207=23, srv613271476=80, srv135363050=18, srv1106275516=4, srv674802528=82, srv877317608=92, srv1039848010=1, srv767258427=89, srv954711736=95, srv1998558817=48, srv812437322=90, srv374136235=60, srv2008955676=50, srv509018606=71, srv1178138356=9, srv1583134907=32, srv1919448122=45, srv1888934480=44, srv1296721447=15, srv95226212=93, srv647946399=81, srv1609135966=35, srv1204619281=11, srv1758125369=41, srv1086051252=3, srv1828014968=42, srv2021037653=51, srv515594154=72, srv1164252757=7, srv1304850798=16, srv1831225009=43, srv1996390770=47, srv1179544135=10, srv2004416148=49, srv416020971=64, srv527201044=73, srv1498813924=25, srv1625481389=38, srv984709116=99, srv100905562=0, srv1464311664=21, srv158584979=33, srv1229603231=13, srv493086728=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-24 23:53:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-24 23:53:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-24 23:53:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-24 23:53:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-24 23:53:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-24 23:53:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-24 23:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-24 23:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-24 23:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-24 23:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-24 23:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-24 23:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-24 23:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-24 23:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-24 23:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-24 23:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-24 23:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-24 23:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-24 23:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-24 23:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-24 23:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-24 23:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-24 23:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-24 23:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-24 23:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-24 23:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-24 23:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-24 23:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-24 23:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-24 23:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-24 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-12-24 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-12-24 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-12-24 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-12-24 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-12-24 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-12-24 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-12-24 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-12-24 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-12-24 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-12-24 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-12-24 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-12-24 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-12-24 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-12-24 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-12-24 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-12-24 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-12-24 23:53:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-12-24 23:53:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-12-24 23:53:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-12-24 23:53:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-12-24 23:53:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-12-24 23:53:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-12-24 23:53:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-12-24 23:53:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-12-24 23:53:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-12-24 23:53:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-12-24 23:53:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-12-24 23:53:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-12-24 23:53:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-12-24 23:53:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-12-24 23:53:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-12-24 23:53:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-12-24 23:53:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-12-24 23:53:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-12-24 23:53:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-12-24 23:53:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-12-24 23:53:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-12-24 23:53:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-12-24 23:53:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-12-24 23:53:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-12-24 23:53:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-12-24 23:53:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-12-24 23:53:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-12-24 23:53:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-12-24 23:53:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-12-24 23:53:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-12-24 23:53:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-12-24 23:53:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-12-24 23:53:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-12-24 23:53:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-12-24 23:53:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-12-24 23:53:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-12-24 23:53:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-12-24 23:53:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-12-24 23:53:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-12-24 23:53:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-12-24 23:53:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-12-24 23:53:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-12-24 23:53:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-12-24 23:53:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-12-24 23:53:23,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-12-24 23:53:23,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-12-24 23:53:23,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-12-24 23:53:23,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-12-24 23:53:23,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-12-24 23:53:23,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-12-24 23:53:23,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-12-24 23:53:23,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-12-24 23:53:23,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-12-24 23:53:23,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-12-24 23:53:24,038 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.6590924490066385E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-24 23:53:24,039 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv2116161037:82 , srv1178138356:83 , srv1474514663:83 , srv1479846093:83 , srv2080530134:83 , srv416245105:83 , srv686443752:83 , srv742629624:83 , srv954711736:83 , srv100905562:84 , srv1086051252:84 , srv1106275516:84 , srv1124773312:84 , srv1171409751:84 , srv1179544135:84 , srv1222202866:84 , srv1271043261:84 , srv1296721447:84 , srv1304850798:84 , srv135363050:84 , srv1464311664:84 , srv1476290207:84 , srv1498813924:84 , srv1540873064:84 , srv1611227535:84 , srv1664413401:84 , srv1831225009:84 , srv1919448122:84 , srv1998558817:84 , srv2021037653:84 , srv2031787984:84 , srv290248788:84 , srv331095736:84 , srv374136235:84 , srv408936987:84 , srv415204158:84 , srv452114297:84 , srv477121615:84 , srv493086728:84 , srv495896058:84 , srv529054392:84 , srv576948304:84 , srv712594253:84 , srv877317608:84 , srv95226212:84 , srv968642543:84 , srv1039848010:85 , srv1068851679:85 , srv1164252757:85 , srv1229603231:85 , srv1310751611:85 , srv1412625480:85 , srv1463924446:85 , srv1499536110:85 , srv1523795877:85 , srv1544707181:85 , srv155288084:85 , srv1569961506:85 , srv1583134907:85 , srv158584979:85 , srv1599324:85 , srv1609135966:85 , srv1613063494:85 , srv1625481389:85 , srv1755361266:85 , srv1758125369:85 , srv1828014968:85 , srv1888934480:85 , srv1996390770:85 , srv2004416148:85 , srv2008955676:85 , srv2113018930:85 , srv2134816287:85 , srv275235570:85 , srv407903732:85 , srv416020971:85 , srv423682277:85 , srv509018606:85 , srv515594154:85 , srv527201044:85 , srv574573162:85 , srv60237876:85 , srv606011854:85 , srv613169750:85 , srv613271476:85 , srv647946399:85 , srv674802528:85 , srv68642599:85 , srv699754202:85 , srv732418113:85 , srv767258427:85 , srv827287294:85 , srv953263816:85 , srv977877422:85 , srv984709116:85 , srv1116029968:86 , srv1204619281:86 , srv1976134831:86 , srv812437322:86 , srv959125770:637 } 2023-12-24 23:53:24,071 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=308 (was 291) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=11153 (was 11751) 2023-12-24 23:53:24,078 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=308, ProcessCount=169, AvailableMemoryMB=11152 2023-12-24 23:53:24,079 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-12-24 23:53:24,079 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-12-24 23:53:24,091 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1046808949:84 , srv1058406244:84 , srv1091337879:84 , srv127004255:84 , srv1290347222:84 , srv1296504647:84 , srv1314747032:84 , srv1421032939:84 , srv1615703360:84 , srv1902328833:84 , srv1912595865:84 , srv2010242849:84 , srv202386485:84 , srv2067569558:84 , srv2136747574:84 , srv2144090742:84 , srv283491599:84 , srv44381749:84 , srv468774402:84 , srv479182066:84 , srv56213039:84 , srv590824541:84 , srv642873849:84 , srv652006371:84 , srv746132191:84 , srv785711268:84 , srv844950202:84 , srv87677484:84 , srv978038230:84 , srv445415392:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-12-24 23:53:24,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-24 23:53:24,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv445415392=18, srv1615703360=8, srv1058406244=1, srv590824541=22, srv1314747032=6, srv127004255=3, srv642873849=23, srv978038230=29, srv785711268=26, srv56213039=21, srv1902328833=9, srv2067569558=13, srv746132191=25, srv2136747574=14, srv87677484=28, srv1912595865=10, srv283491599=16, srv1290347222=4, srv2144090742=15, srv1046808949=0, srv44381749=17, srv479182066=20, srv1091337879=2, srv652006371=24, srv1296504647=5, srv1421032939=7, srv2010242849=11, srv468774402=19, srv202386485=12, srv844950202=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-24 23:53:24,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-24 23:53:24,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-24 23:53:24,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-24 23:53:24,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-24 23:53:24,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-24 23:53:24,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-24 23:53:24,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-24 23:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-24 23:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-24 23:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-24 23:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-24 23:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-24 23:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-24 23:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-24 23:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-24 23:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-24 23:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-24 23:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-24 23:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-24 23:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-24 23:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-24 23:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-24 23:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-24 23:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-24 23:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-24 23:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-24 23:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-24 23:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-24 23:53:24,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-24 23:53:24,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-24 23:53:24,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-24 23:53:24,116 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1158900166567693, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-24 23:53:24,117 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912558208922608, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1158900166567693, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-12-24 23:55:24,106 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5539125 different iterations. Found a solution that moves 1860 regions; Going from a computed imbalance of 0.9912558208922608 to a new imbalance of 4.341332789166274E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6888888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-24 23:55:24,112 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1046808949:90 , srv1058406244:90 , srv1091337879:90 , srv127004255:90 , srv1290347222:90 , srv1296504647:90 , srv1314747032:90 , srv1421032939:90 , srv1615703360:90 , srv1902328833:90 , srv1912595865:90 , srv2010242849:90 , srv202386485:90 , srv2067569558:90 , srv2136747574:90 , srv2144090742:90 , srv283491599:90 , srv44381749:90 , srv445415392:90 , srv468774402:90 , srv479182066:90 , srv56213039:90 , srv590824541:90 , srv642873849:90 , srv652006371:90 , srv746132191:90 , srv785711268:90 , srv844950202:90 , srv87677484:90 , srv978038230:90 } 2023-12-24 23:55:24,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-12-24 23:55:24,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv445415392=18, srv1615703360=8, srv1058406244=1, srv590824541=22, srv1314747032=6, srv127004255=3, srv642873849=23, srv978038230=29, srv785711268=26, srv56213039=21, srv1902328833=9, srv2067569558=13, srv746132191=25, srv2136747574=14, srv87677484=28, srv1912595865=10, srv283491599=16, srv1290347222=4, srv2144090742=15, srv1046808949=0, srv44381749=17, srv479182066=20, srv1091337879=2, srv652006371=24, srv1296504647=5, srv1421032939=7, srv2010242849=11, srv468774402=19, srv202386485=12, srv844950202=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-12-24 23:55:24,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-12-24 23:55:24,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-12-24 23:55:24,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-12-24 23:55:24,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-12-24 23:55:24,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-12-24 23:55:24,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-12-24 23:55:24,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-12-24 23:55:24,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-12-24 23:55:24,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-12-24 23:55:24,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-12-24 23:55:24,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-12-24 23:55:24,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-12-24 23:55:24,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-12-24 23:55:24,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-12-24 23:55:24,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-12-24 23:55:24,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-12-24 23:55:24,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-12-24 23:55:24,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-12-24 23:55:24,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-12-24 23:55:24,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-12-24 23:55:24,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-12-24 23:55:24,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-12-24 23:55:24,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-12-24 23:55:24,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-12-24 23:55:24,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-12-24 23:55:24,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-12-24 23:55:24,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-12-24 23:55:24,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-12-24 23:55:24,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-12-24 23:55:24,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-12-24 23:55:24,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-12-24 23:55:24,137 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-12-24 23:55:24,137 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1046808949:90 , srv1058406244:90 , srv1091337879:90 , srv127004255:90 , srv1290347222:90 , srv1296504647:90 , srv1314747032:90 , srv1421032939:90 , srv1615703360:90 , srv1902328833:90 , srv1912595865:90 , srv2010242849:90 , srv202386485:90 , srv2067569558:90 , srv2136747574:90 , srv2144090742:90 , srv283491599:90 , srv44381749:90 , srv445415392:90 , srv468774402:90 , srv479182066:90 , srv56213039:90 , srv590824541:90 , srv642873849:90 , srv652006371:90 , srv746132191:90 , srv785711268:90 , srv844950202:90 , srv87677484:90 , srv978038230:90 } 2023-12-24 23:55:24,153 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=225 (was 308), ProcessCount=170 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=10675 (was 11152)