2023-10-19 18:53:27,521 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-10-19 18:53:27,821 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-19 18:53:27,933 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-10-19 18:53:28,001 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-19 18:53:28,009 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-19 18:53:28,024 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=310, ProcessCount=169, AvailableMemoryMB=11264 2023-10-19 18:53:28,026 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-19 18:53:28,027 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-19 18:53:28,161 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1017359666:84 , srv1031917979:84 , srv1066424162:84 , srv1074541621:84 , srv1098097628:84 , srv1099175580:84 , srv1111521139:84 , srv1141125411:84 , srv115022089:84 , srv1159388076:84 , srv1173127194:84 , srv1181639057:84 , srv1189134801:84 , srv1195393379:84 , srv1238700620:84 , srv1241989146:84 , srv1283382444:84 , srv1303809273:84 , srv1309637023:84 , srv1313568469:84 , srv1344432469:84 , srv1350517833:84 , srv1363600636:84 , srv1376921598:84 , srv1379235248:84 , srv1413689296:84 , srv1440265411:84 , srv146240685:84 , srv1477071856:84 , srv1516805281:84 , srv152443299:84 , srv1543910622:84 , srv1562672268:84 , srv159938100:84 , srv1617235643:84 , srv1680307410:84 , srv1683316709:84 , srv1695943230:84 , srv1697736993:84 , srv1707700838:84 , srv1748314348:84 , srv1748578283:84 , srv1797830770:84 , srv1849798095:84 , srv1870359988:84 , srv1875302350:84 , srv1909533388:84 , srv1935056484:84 , srv1947279920:84 , srv2024177023:84 , srv2036420180:84 , srv2043419130:84 , srv2043465693:84 , srv2052481948:84 , srv2057246221:84 , srv2077732308:84 , srv2087594516:84 , srv2093823164:84 , srv2095770785:84 , srv210983931:84 , srv2138017684:84 , srv223530145:84 , srv257950480:84 , srv267198752:84 , srv300081550:84 , srv304676882:84 , srv331349209:84 , srv355508678:84 , srv358112045:84 , srv396621168:84 , srv399652278:84 , srv45609493:84 , srv465208772:84 , srv483266234:84 , srv51961111:84 , srv567836301:84 , srv595812837:84 , srv617569491:84 , srv62831935:84 , srv66271521:84 , srv664716809:84 , srv696773079:84 , srv735272239:84 , srv73879462:84 , srv76309603:84 , srv771600996:84 , srv824547989:84 , srv841122059:84 , srv855347983:84 , srv871571782:84 , srv875179792:84 , srv878487320:84 , srv889895391:84 , srv9278997:84 , srv930856481:84 , srv966133389:84 , srv969532819:84 , srv982238849:84 , srv996920502:84 , srv786483626:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-10-19 18:53:28,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 18:53:28,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1875302350=45, srv1173127194=10, srv1617235643=34, srv1562672268=32, srv595812837=76, srv567836301=75, srv1413689296=25, srv1159388076=9, srv1017359666=0, srv1283382444=16, srv2077732308=55, srv875179792=91, srv1935056484=47, srv1309637023=18, srv930856481=95, srv1909533388=46, srv1748578283=41, srv1181639057=11, srv966133389=96, srv1683316709=36, srv399652278=70, srv159938100=33, srv2057246221=54, srv1543910622=31, srv2024177023=49, srv2036420180=50, srv855347983=89, srv735272239=82, srv2043465693=52, srv1350517833=21, srv210983931=59, srv62831935=78, srv2093823164=57, srv465208772=72, srv1313568469=19, srv996920502=99, srv617569491=77, srv664716809=80, srv841122059=88, srv1099175580=5, srv358112045=68, srv1241989146=15, srv257950480=62, srv1066424162=2, srv1344432469=20, srv2138017684=60, srv223530145=61, srv1074541621=3, srv300081550=64, srv1141125411=7, srv1195393379=13, srv73879462=83, srv1707700838=39, srv1238700620=14, srv267198752=63, srv2052481948=53, srv51961111=74, srv1947279920=48, srv1748314348=40, srv2043419130=51, srv1849798095=43, srv1516805281=29, srv771600996=85, srv66271521=79, srv1797830770=42, srv1363600636=22, srv1189134801=12, srv1379235248=24, srv355508678=67, srv45609493=71, srv331349209=66, srv1440265411=26, srv1303809273=17, srv9278997=94, srv696773079=81, srv871571782=90, srv396621168=69, srv1098097628=4, srv146240685=27, srv1376921598=23, srv2087594516=56, srv1697736993=38, srv483266234=73, srv982238849=98, srv1031917979=1, srv889895391=93, srv76309603=84, srv2095770785=58, srv115022089=8, srv969532819=97, srv1111521139=6, srv786483626=86, srv152443299=30, srv1695943230=37, srv1870359988=44, srv824547989=87, srv1477071856=28, srv304676882=65, srv1680307410=35, srv878487320=92} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 18:53:28,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 18:53:28,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 18:53:28,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 18:53:28,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 18:53:28,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 18:53:28,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 18:53:28,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 18:53:28,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 18:53:28,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 18:53:28,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 18:53:28,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 18:53:28,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 18:53:28,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 18:53:28,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 18:53:28,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 18:53:28,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 18:53:28,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 18:53:28,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 18:53:28,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 18:53:28,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 18:53:28,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 18:53:28,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 18:53:28,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 18:53:28,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 18:53:28,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 18:53:28,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 18:53:28,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 18:53:28,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 18:53:28,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 18:53:28,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 18:53:28,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-19 18:53:28,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-19 18:53:28,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-19 18:53:28,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-19 18:53:28,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-19 18:53:28,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-19 18:53:28,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-19 18:53:28,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-19 18:53:28,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-19 18:53:28,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-19 18:53:28,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-19 18:53:28,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-19 18:53:28,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-19 18:53:28,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-19 18:53:28,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-19 18:53:28,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-19 18:53:28,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-19 18:53:28,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-19 18:53:28,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-19 18:53:28,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-19 18:53:28,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-19 18:53:28,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-19 18:53:28,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-19 18:53:28,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-19 18:53:28,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-19 18:53:28,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-19 18:53:28,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-19 18:53:28,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-19 18:53:28,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-19 18:53:28,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-19 18:53:28,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-19 18:53:28,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-19 18:53:28,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-19 18:53:28,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-19 18:53:28,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-19 18:53:28,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-19 18:53:28,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-19 18:53:28,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-19 18:53:28,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-19 18:53:28,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-19 18:53:28,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-19 18:53:28,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-19 18:53:28,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-19 18:53:28,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-19 18:53:28,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-19 18:53:28,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-19 18:53:28,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-19 18:53:28,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-19 18:53:28,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-19 18:53:28,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-19 18:53:28,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-19 18:53:28,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-19 18:53:28,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-19 18:53:28,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-19 18:53:28,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-19 18:53:28,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-19 18:53:28,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-19 18:53:28,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-19 18:53:28,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-19 18:53:28,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-19 18:53:28,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-19 18:53:28,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-19 18:53:28,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-19 18:53:28,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-19 18:53:28,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-19 18:53:28,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-19 18:53:28,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-19 18:53:28,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-19 18:53:28,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-19 18:53:28,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-19 18:53:28,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-19 18:53:28,489 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 18:53:28,489 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-19 18:53:28,490 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-19 18:53:28,882 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 585 ms to try 5000 different iterations. Found a solution that moves 4683 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1848786374487991. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041414989999792075, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.061596784500739375, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5203333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041414989999792075, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18016666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24625, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 18:53:28,901 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1283382444:81 , srv1099175580:82 , srv1195393379:82 , srv1303809273:82 , srv2077732308:82 , srv1683316709:83 , srv2043465693:83 , srv1066424162:84 , srv1098097628:84 , srv1159388076:84 , srv1344432469:84 , srv1413689296:84 , srv1516805281:84 , srv1617235643:84 , srv1695943230:84 , srv1697736993:84 , srv1707700838:84 , srv1748578283:84 , srv1870359988:84 , srv1875302350:84 , srv2052481948:84 , srv2138017684:84 , srv257950480:84 , srv300081550:84 , srv358112045:84 , srv878487320:84 , srv1031917979:85 , srv1074541621:85 , srv1173127194:85 , srv1189134801:85 , srv146240685:85 , srv1748314348:85 , srv1797830770:85 , srv1909533388:85 , srv2043419130:85 , srv2087594516:85 , srv304676882:85 , srv465208772:85 , srv62831935:85 , srv73879462:85 , srv771600996:85 , srv871571782:85 , srv875179792:85 , srv1141125411:86 , srv115022089:86 , srv1238700620:86 , srv1440265411:86 , srv1477071856:86 , srv159938100:86 , srv1680307410:86 , srv1849798095:86 , srv1935056484:86 , srv2036420180:86 , srv355508678:86 , srv45609493:86 , srv567836301:86 , srv595812837:86 , srv824547989:86 , srv841122059:86 , srv966133389:86 , srv969532819:86 , srv996920502:86 , srv1313568469:87 , srv1379235248:87 , srv1562672268:87 , srv2057246221:87 , srv223530145:87 , srv396621168:87 , srv664716809:87 , srv735272239:87 , srv76309603:87 , srv930856481:87 , srv1111521139:88 , srv1181639057:88 , srv1350517833:88 , srv1363600636:88 , srv152443299:88 , srv1947279920:88 , srv2024177023:88 , srv2093823164:88 , srv210983931:88 , srv267198752:88 , srv331349209:88 , srv66271521:88 , srv696773079:88 , srv982238849:88 , srv1241989146:89 , srv1376921598:89 , srv855347983:89 , srv1017359666:90 , srv617569491:90 , srv1543910622:91 , srv399652278:91 , srv483266234:91 , srv1309637023:92 , srv2095770785:92 , srv9278997:92 , srv889895391:95 , srv51961111:99 , srv786483626:458 } 2023-10-19 18:53:28,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 18:53:28,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1875302350=45, srv1173127194=10, srv1617235643=34, srv1562672268=32, srv595812837=76, srv567836301=75, srv1413689296=25, srv1159388076=9, srv1017359666=0, srv1283382444=16, srv2077732308=55, srv875179792=91, srv1935056484=47, srv1309637023=18, srv930856481=95, srv1909533388=46, srv1748578283=41, srv1181639057=11, srv966133389=96, srv1683316709=36, srv399652278=70, srv159938100=33, srv2057246221=54, srv1543910622=31, srv2024177023=49, srv2036420180=50, srv855347983=89, srv735272239=82, srv2043465693=52, srv1350517833=21, srv210983931=59, srv62831935=78, srv2093823164=57, srv465208772=72, srv1313568469=19, srv996920502=99, srv617569491=77, srv664716809=80, srv841122059=88, srv1099175580=5, srv358112045=68, srv1241989146=15, srv257950480=62, srv1066424162=2, srv1344432469=20, srv2138017684=60, srv223530145=61, srv1074541621=3, srv300081550=64, srv1141125411=7, srv1195393379=13, srv73879462=83, srv1707700838=39, srv1238700620=14, srv267198752=63, srv2052481948=53, srv51961111=74, srv1947279920=48, srv1748314348=40, srv2043419130=51, srv1849798095=43, srv1516805281=29, srv771600996=85, srv66271521=79, srv1797830770=42, srv1363600636=22, srv1189134801=12, srv1379235248=24, srv355508678=67, srv45609493=71, srv331349209=66, srv1440265411=26, srv1303809273=17, srv9278997=94, srv696773079=81, srv871571782=90, srv396621168=69, srv1098097628=4, srv146240685=27, srv1376921598=23, srv2087594516=56, srv1697736993=38, srv483266234=73, srv982238849=98, srv1031917979=1, srv889895391=93, srv76309603=84, srv2095770785=58, srv115022089=8, srv969532819=97, srv1111521139=6, srv786483626=86, srv152443299=30, srv1695943230=37, srv1870359988=44, srv824547989=87, srv1477071856=28, srv304676882=65, srv1680307410=35, srv878487320=92} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 18:53:28,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 18:53:28,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 18:53:28,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 18:53:28,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 18:53:28,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 18:53:28,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 18:53:28,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 18:53:28,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 18:53:28,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 18:53:28,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 18:53:28,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 18:53:28,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 18:53:28,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 18:53:28,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 18:53:28,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 18:53:28,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 18:53:28,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 18:53:28,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 18:53:28,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 18:53:28,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 18:53:28,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 18:53:28,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 18:53:28,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 18:53:28,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 18:53:28,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 18:53:28,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 18:53:28,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 18:53:28,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 18:53:28,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 18:53:28,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 18:53:28,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-19 18:53:28,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-19 18:53:28,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-19 18:53:28,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-19 18:53:28,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-19 18:53:28,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-19 18:53:28,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-19 18:53:28,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-19 18:53:28,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-19 18:53:28,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-19 18:53:28,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-19 18:53:28,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-19 18:53:28,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-19 18:53:28,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-19 18:53:28,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-19 18:53:28,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-19 18:53:28,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-19 18:53:28,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-19 18:53:28,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-19 18:53:28,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-19 18:53:28,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-19 18:53:28,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-19 18:53:28,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-19 18:53:28,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-19 18:53:28,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-19 18:53:28,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-19 18:53:28,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-19 18:53:28,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-19 18:53:28,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-19 18:53:28,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-19 18:53:28,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-19 18:53:28,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-19 18:53:28,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-19 18:53:28,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-19 18:53:28,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-19 18:53:28,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-19 18:53:28,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-19 18:53:28,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-19 18:53:28,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-19 18:53:28,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-19 18:53:28,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-19 18:53:28,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-19 18:53:28,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-19 18:53:28,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-19 18:53:28,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-19 18:53:28,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-19 18:53:28,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-19 18:53:28,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-19 18:53:28,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-19 18:53:28,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-19 18:53:28,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-19 18:53:28,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-19 18:53:28,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-19 18:53:28,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-19 18:53:28,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-19 18:53:28,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-19 18:53:28,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-19 18:53:28,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-19 18:53:28,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-19 18:53:28,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-19 18:53:28,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-19 18:53:28,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-19 18:53:28,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-19 18:53:28,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-19 18:53:28,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-19 18:53:28,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-19 18:53:28,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-19 18:53:28,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-19 18:53:28,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-19 18:53:28,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-19 18:53:28,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-19 18:53:29,073 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041414989999792075, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.061596784500739375, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041414989999792075, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18016666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24625, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 18:53:29,074 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-19 18:53:29,074 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18484584638194157, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041414989999792075, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.061596784500739375, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041414989999792075, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18016666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24625, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-19 18:53:29,354 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 408 ms to try 5000 different iterations. Found a solution that moves 2625 regions; Going from a computed imbalance of 0.18484584638194157 to a new imbalance of 0.016284017483093615. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023189928624828524); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04032972964394579, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2916666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023189928624828524); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01325); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04491666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 18:53:29,364 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2095770785:78 , srv51961111:78 , srv1017359666:81 , srv1376921598:81 , srv399652278:81 , srv483266234:81 , srv889895391:81 , srv1181639057:82 , srv1241989146:82 , srv1309637023:82 , srv1350517833:82 , srv1363600636:82 , srv331349209:82 , srv9278997:82 , srv1379235248:83 , srv152443299:83 , srv1543910622:83 , srv1935056484:83 , srv2024177023:83 , srv210983931:83 , srv396621168:83 , srv617569491:83 , srv1031917979:84 , srv115022089:84 , srv159938100:84 , srv1849798095:84 , srv2057246221:84 , srv2093823164:84 , srv267198752:84 , srv66271521:84 , srv664716809:84 , srv1562672268:85 , srv1680307410:85 , srv1947279920:85 , srv2036420180:85 , srv2138017684:85 , srv595812837:85 , srv696773079:85 , srv771600996:85 , srv841122059:85 , srv871571782:85 , srv930856481:85 , srv996920502:85 , srv1111521139:86 , srv1173127194:86 , srv1238700620:86 , srv1313568469:86 , srv1617235643:86 , srv2043419130:86 , srv223530145:86 , srv304676882:86 , srv567836301:86 , srv62831935:86 , srv73879462:86 , srv76309603:86 , srv824547989:86 , srv875179792:86 , srv1074541621:87 , srv1098097628:87 , srv1189134801:87 , srv1516805281:87 , srv1695943230:87 , srv1707700838:87 , srv1748314348:87 , srv1797830770:87 , srv2087594516:87 , srv355508678:87 , srv45609493:87 , srv465208772:87 , srv735272239:87 , srv855347983:87 , srv878487320:87 , srv966133389:87 , srv969532819:87 , srv1141125411:88 , srv1159388076:88 , srv1303809273:88 , srv1413689296:88 , srv1440265411:88 , srv1477071856:88 , srv1683316709:88 , srv1875302350:88 , srv1909533388:88 , srv358112045:88 , srv982238849:88 , srv1066424162:89 , srv1099175580:89 , srv1195393379:89 , srv1344432469:89 , srv146240685:89 , srv1748578283:89 , srv1870359988:89 , srv2043465693:89 , srv2077732308:89 , srv1283382444:90 , srv1697736993:90 , srv2052481948:90 , srv300081550:90 , srv257950480:91 , srv786483626:522 } 2023-10-19 18:53:29,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 18:53:29,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1875302350=45, srv1173127194=10, srv1617235643=34, srv1562672268=32, srv595812837=76, srv567836301=75, srv1413689296=25, srv1159388076=9, srv1017359666=0, srv1283382444=16, srv2077732308=55, srv875179792=91, srv1935056484=47, srv1309637023=18, srv930856481=95, srv1909533388=46, srv1748578283=41, srv1181639057=11, srv966133389=96, srv1683316709=36, srv399652278=70, srv159938100=33, srv2057246221=54, srv1543910622=31, srv2024177023=49, srv2036420180=50, srv855347983=89, srv735272239=82, srv2043465693=52, srv1350517833=21, srv210983931=59, srv62831935=78, srv2093823164=57, srv465208772=72, srv1313568469=19, srv996920502=99, srv617569491=77, srv664716809=80, srv841122059=88, srv1099175580=5, srv358112045=68, srv1241989146=15, srv257950480=62, srv1066424162=2, srv1344432469=20, srv2138017684=60, srv223530145=61, srv1074541621=3, srv300081550=64, srv1141125411=7, srv1195393379=13, srv73879462=83, srv1707700838=39, srv1238700620=14, srv267198752=63, srv2052481948=53, srv51961111=74, srv1947279920=48, srv1748314348=40, srv2043419130=51, srv1849798095=43, srv1516805281=29, srv771600996=85, srv66271521=79, srv1797830770=42, srv1363600636=22, srv1189134801=12, srv1379235248=24, srv355508678=67, srv45609493=71, srv331349209=66, srv1440265411=26, srv1303809273=17, srv9278997=94, srv696773079=81, srv871571782=90, srv396621168=69, srv1098097628=4, srv146240685=27, srv1376921598=23, srv2087594516=56, srv1697736993=38, srv483266234=73, srv982238849=98, srv1031917979=1, srv889895391=93, srv76309603=84, srv2095770785=58, srv115022089=8, srv969532819=97, srv1111521139=6, srv786483626=86, srv152443299=30, srv1695943230=37, srv1870359988=44, srv824547989=87, srv1477071856=28, srv304676882=65, srv1680307410=35, srv878487320=92} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 18:53:29,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 18:53:29,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 18:53:29,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 18:53:29,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 18:53:29,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 18:53:29,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 18:53:29,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 18:53:29,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 18:53:29,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 18:53:29,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 18:53:29,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 18:53:29,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 18:53:29,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 18:53:29,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 18:53:29,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 18:53:29,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 18:53:29,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 18:53:29,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 18:53:29,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 18:53:29,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 18:53:29,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 18:53:29,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 18:53:29,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 18:53:29,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 18:53:29,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 18:53:29,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 18:53:29,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 18:53:29,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 18:53:29,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 18:53:29,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 18:53:29,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-19 18:53:29,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-19 18:53:29,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-19 18:53:29,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-19 18:53:29,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-19 18:53:29,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-19 18:53:29,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-19 18:53:29,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-19 18:53:29,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-19 18:53:29,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-19 18:53:29,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-19 18:53:29,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-19 18:53:29,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-19 18:53:29,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-19 18:53:29,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-19 18:53:29,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-19 18:53:29,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-19 18:53:29,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-19 18:53:29,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-19 18:53:29,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-19 18:53:29,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-19 18:53:29,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-19 18:53:29,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-19 18:53:29,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-19 18:53:29,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-19 18:53:29,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-19 18:53:29,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-19 18:53:29,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-19 18:53:29,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-19 18:53:29,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-19 18:53:29,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-19 18:53:29,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-19 18:53:29,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-19 18:53:29,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-19 18:53:29,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-19 18:53:29,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-19 18:53:29,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-19 18:53:29,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-19 18:53:29,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-19 18:53:29,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-19 18:53:29,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-19 18:53:29,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-19 18:53:29,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-19 18:53:29,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-19 18:53:29,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-19 18:53:29,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-19 18:53:29,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-19 18:53:29,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-19 18:53:29,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-19 18:53:29,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-19 18:53:29,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-19 18:53:29,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-19 18:53:29,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-19 18:53:29,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-19 18:53:29,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-19 18:53:29,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-19 18:53:29,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-19 18:53:29,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-19 18:53:29,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-19 18:53:29,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-19 18:53:29,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-19 18:53:29,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-19 18:53:29,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-19 18:53:29,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-19 18:53:29,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-19 18:53:29,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-19 18:53:29,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-19 18:53:29,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-19 18:53:29,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-19 18:53:29,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-19 18:53:29,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-19 18:53:29,517 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023189928624828524); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04032972964394579, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023189928624828524); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01325); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04491666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 18:53:29,517 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-19 18:53:29,517 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01626563684023626, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023189928624828524); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04032972964394579, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023189928624828524); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01325); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04491666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-19 18:53:29,816 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 397 ms to try 5000 different iterations. Found a solution that moves 1014 regions; Going from a computed imbalance of 0.01626563684023626 to a new imbalance of 7.573717573143419E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007899480725663495); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011556720997950364); MoveCostFunction : (multiplier=7.0, imbalance=0.11266666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007899480725663495); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 18:53:29,820 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv855347983:81 , srv889895391:81 , srv1111521139:82 , srv257950480:82 , srv51961111:82 , srv982238849:82 , srv1309637023:83 , srv1477071856:83 , srv1543910622:83 , srv1697736993:83 , srv735272239:83 , srv76309603:83 , srv1066424162:84 , srv1141125411:84 , srv1344432469:84 , srv1440265411:84 , srv146240685:84 , srv1870359988:84 , srv2052481948:84 , srv300081550:84 , srv355508678:84 , srv45609493:84 , srv465208772:84 , srv617569491:84 , srv9278997:84 , srv966133389:84 , srv1074541621:85 , srv1098097628:85 , srv1159388076:85 , srv1189134801:85 , srv1238700620:85 , srv1516805281:85 , srv1707700838:85 , srv1748314348:85 , srv1748578283:85 , srv1875302350:85 , srv1909533388:85 , srv1947279920:85 , srv2043465693:85 , srv210983931:85 , srv223530145:85 , srv358112045:85 , srv399652278:85 , srv567836301:85 , srv696773079:85 , srv73879462:85 , srv875179792:85 , srv969532819:85 , srv1017359666:86 , srv1099175580:86 , srv1173127194:86 , srv1181639057:86 , srv1195393379:86 , srv1283382444:86 , srv1313568469:86 , srv1363600636:86 , srv1413689296:86 , srv152443299:86 , srv1680307410:86 , srv1683316709:86 , srv1695943230:86 , srv1797830770:86 , srv2024177023:86 , srv2036420180:86 , srv2043419130:86 , srv2057246221:86 , srv2093823164:86 , srv267198752:86 , srv483266234:86 , srv66271521:86 , srv824547989:86 , srv841122059:86 , srv878487320:86 , srv930856481:86 , srv1031917979:87 , srv115022089:87 , srv1241989146:87 , srv1303809273:87 , srv1350517833:87 , srv1376921598:87 , srv1562672268:87 , srv159938100:87 , srv1617235643:87 , srv1849798095:87 , srv2077732308:87 , srv2087594516:87 , srv2095770785:87 , srv331349209:87 , srv595812837:87 , srv62831935:87 , srv664716809:87 , srv771600996:87 , srv871571782:87 , srv996920502:87 , srv1379235248:88 , srv304676882:88 , srv396621168:88 , srv1935056484:89 , srv2138017684:89 , srv786483626:548 } 2023-10-19 18:53:29,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 18:53:29,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1875302350=45, srv1173127194=10, srv1617235643=34, srv1562672268=32, srv595812837=76, srv567836301=75, srv1413689296=25, srv1159388076=9, srv1017359666=0, srv1283382444=16, srv2077732308=55, srv875179792=91, srv1935056484=47, srv1309637023=18, srv930856481=95, srv1909533388=46, srv1748578283=41, srv1181639057=11, srv966133389=96, srv1683316709=36, srv399652278=70, srv159938100=33, srv2057246221=54, srv1543910622=31, srv2024177023=49, srv2036420180=50, srv855347983=89, srv735272239=82, srv2043465693=52, srv1350517833=21, srv210983931=59, srv62831935=78, srv2093823164=57, srv465208772=72, srv1313568469=19, srv996920502=99, srv617569491=77, srv664716809=80, srv841122059=88, srv1099175580=5, srv358112045=68, srv1241989146=15, srv257950480=62, srv1066424162=2, srv1344432469=20, srv2138017684=60, srv223530145=61, srv1074541621=3, srv300081550=64, srv1141125411=7, srv1195393379=13, srv73879462=83, srv1707700838=39, srv1238700620=14, srv267198752=63, srv2052481948=53, srv51961111=74, srv1947279920=48, srv1748314348=40, srv2043419130=51, srv1849798095=43, srv1516805281=29, srv771600996=85, srv66271521=79, srv1797830770=42, srv1363600636=22, srv1189134801=12, srv1379235248=24, srv355508678=67, srv45609493=71, srv331349209=66, srv1440265411=26, srv1303809273=17, srv9278997=94, srv696773079=81, srv871571782=90, srv396621168=69, srv1098097628=4, srv146240685=27, srv1376921598=23, srv2087594516=56, srv1697736993=38, srv483266234=73, srv982238849=98, srv1031917979=1, srv889895391=93, srv76309603=84, srv2095770785=58, srv115022089=8, srv969532819=97, srv1111521139=6, srv786483626=86, srv152443299=30, srv1695943230=37, srv1870359988=44, srv824547989=87, srv1477071856=28, srv304676882=65, srv1680307410=35, srv878487320=92} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 18:53:29,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 18:53:29,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 18:53:29,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 18:53:29,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 18:53:29,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 18:53:29,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 18:53:29,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 18:53:29,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 18:53:29,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 18:53:29,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 18:53:29,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 18:53:29,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 18:53:29,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 18:53:29,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 18:53:29,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 18:53:29,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 18:53:29,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 18:53:29,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 18:53:29,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 18:53:29,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 18:53:29,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 18:53:29,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 18:53:29,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 18:53:29,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 18:53:29,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 18:53:29,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 18:53:29,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 18:53:29,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 18:53:29,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 18:53:29,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 18:53:29,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-19 18:53:29,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-19 18:53:29,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-19 18:53:29,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-19 18:53:29,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-19 18:53:29,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-19 18:53:29,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-19 18:53:29,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-19 18:53:29,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-19 18:53:29,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-19 18:53:29,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-19 18:53:29,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-19 18:53:29,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-19 18:53:29,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-19 18:53:29,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-19 18:53:29,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-19 18:53:29,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-19 18:53:29,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-19 18:53:29,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-19 18:53:29,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-19 18:53:29,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-19 18:53:29,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-19 18:53:29,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-19 18:53:29,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-19 18:53:29,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-19 18:53:29,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-19 18:53:29,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-19 18:53:29,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-19 18:53:29,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-19 18:53:29,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-19 18:53:29,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-19 18:53:29,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-19 18:53:29,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-19 18:53:29,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-19 18:53:29,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-19 18:53:29,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-19 18:53:29,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-19 18:53:29,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-19 18:53:29,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-19 18:53:29,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-19 18:53:29,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-19 18:53:29,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-19 18:53:29,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-19 18:53:29,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-19 18:53:29,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-19 18:53:29,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-19 18:53:29,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-19 18:53:29,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-19 18:53:29,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-19 18:53:29,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-19 18:53:29,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-19 18:53:29,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-19 18:53:29,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-19 18:53:29,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-19 18:53:29,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-19 18:53:29,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-19 18:53:29,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-19 18:53:29,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-19 18:53:29,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-19 18:53:29,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-19 18:53:29,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-19 18:53:29,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-19 18:53:29,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-19 18:53:29,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-19 18:53:29,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-19 18:53:29,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-19 18:53:29,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-19 18:53:29,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-19 18:53:29,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-19 18:53:29,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-19 18:53:29,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-19 18:53:30,012 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007899480725663495); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011556720997950364); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007899480725663495); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 18:53:30,012 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-19 18:53:30,013 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.50271577559157E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007899480725663495); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011556720997950364); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007899480725663495); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-19 18:53:30,455 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 582 ms to try 5000 different iterations. Found a solution that moves 294 regions; Going from a computed imbalance of 7.50271577559157E-4 to a new imbalance of 7.012674990179417E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002920572180767871); MoveCostFunction : (multiplier=7.0, imbalance=0.03266666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.833333333333334E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 18:53:30,456 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1313568469:83 , srv1562672268:83 , srv2087594516:83 , srv304676882:83 , srv51961111:83 , srv62831935:83 , srv1066424162:84 , srv1141125411:84 , srv1241989146:84 , srv1309637023:84 , srv1379235248:84 , srv1413689296:84 , srv1440265411:84 , srv146240685:84 , srv152443299:84 , srv1683316709:84 , srv1697736993:84 , srv1748578283:84 , srv1797830770:84 , srv1909533388:84 , srv1935056484:84 , srv1947279920:84 , srv2052481948:84 , srv2057246221:84 , srv2138017684:84 , srv267198752:84 , srv300081550:84 , srv396621168:84 , srv399652278:84 , srv483266234:84 , srv595812837:84 , srv66271521:84 , srv664716809:84 , srv696773079:84 , srv9278997:84 , srv930856481:84 , srv969532819:84 , srv982238849:84 , srv996920502:84 , srv1017359666:85 , srv1074541621:85 , srv1098097628:85 , srv1099175580:85 , srv115022089:85 , srv1159388076:85 , srv1173127194:85 , srv1189134801:85 , srv1238700620:85 , srv1283382444:85 , srv1303809273:85 , srv1344432469:85 , srv1350517833:85 , srv1376921598:85 , srv1477071856:85 , srv1543910622:85 , srv159938100:85 , srv1617235643:85 , srv1680307410:85 , srv1695943230:85 , srv1748314348:85 , srv1849798095:85 , srv1870359988:85 , srv1875302350:85 , srv2024177023:85 , srv2036420180:85 , srv2043465693:85 , srv2077732308:85 , srv2093823164:85 , srv2095770785:85 , srv223530145:85 , srv257950480:85 , srv331349209:85 , srv355508678:85 , srv45609493:85 , srv567836301:85 , srv617569491:85 , srv735272239:85 , srv73879462:85 , srv824547989:85 , srv841122059:85 , srv855347983:85 , srv871571782:85 , srv878487320:85 , srv966133389:85 , srv1031917979:86 , srv1111521139:86 , srv1181639057:86 , srv1195393379:86 , srv1363600636:86 , srv1516805281:86 , srv1707700838:86 , srv2043419130:86 , srv358112045:86 , srv465208772:86 , srv76309603:86 , srv771600996:86 , srv875179792:86 , srv889895391:86 , srv210983931:87 , srv786483626:614 } 2023-10-19 18:53:30,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 18:53:30,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1875302350=45, srv1173127194=10, srv1617235643=34, srv1562672268=32, srv595812837=76, srv567836301=75, srv1413689296=25, srv1159388076=9, srv1017359666=0, srv1283382444=16, srv2077732308=55, srv875179792=91, srv1935056484=47, srv1309637023=18, srv930856481=95, srv1909533388=46, srv1748578283=41, srv1181639057=11, srv966133389=96, srv1683316709=36, srv399652278=70, srv159938100=33, srv2057246221=54, srv1543910622=31, srv2024177023=49, srv2036420180=50, srv855347983=89, srv735272239=82, srv2043465693=52, srv1350517833=21, srv210983931=59, srv62831935=78, srv2093823164=57, srv465208772=72, srv1313568469=19, srv996920502=99, srv617569491=77, srv664716809=80, srv841122059=88, srv1099175580=5, srv358112045=68, srv1241989146=15, srv257950480=62, srv1066424162=2, srv1344432469=20, srv2138017684=60, srv223530145=61, srv1074541621=3, srv300081550=64, srv1141125411=7, srv1195393379=13, srv73879462=83, srv1707700838=39, srv1238700620=14, srv267198752=63, srv2052481948=53, srv51961111=74, srv1947279920=48, srv1748314348=40, srv2043419130=51, srv1849798095=43, srv1516805281=29, srv771600996=85, srv66271521=79, srv1797830770=42, srv1363600636=22, srv1189134801=12, srv1379235248=24, srv355508678=67, srv45609493=71, srv331349209=66, srv1440265411=26, srv1303809273=17, srv9278997=94, srv696773079=81, srv871571782=90, srv396621168=69, srv1098097628=4, srv146240685=27, srv1376921598=23, srv2087594516=56, srv1697736993=38, srv483266234=73, srv982238849=98, srv1031917979=1, srv889895391=93, srv76309603=84, srv2095770785=58, srv115022089=8, srv969532819=97, srv1111521139=6, srv786483626=86, srv152443299=30, srv1695943230=37, srv1870359988=44, srv824547989=87, srv1477071856=28, srv304676882=65, srv1680307410=35, srv878487320=92} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 18:53:30,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 18:53:30,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 18:53:30,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 18:53:30,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 18:53:30,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 18:53:30,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 18:53:30,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 18:53:30,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 18:53:30,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 18:53:30,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 18:53:30,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 18:53:30,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 18:53:30,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 18:53:30,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 18:53:30,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 18:53:30,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 18:53:30,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 18:53:30,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 18:53:30,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 18:53:30,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 18:53:30,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 18:53:30,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 18:53:30,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 18:53:30,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 18:53:30,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 18:53:30,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 18:53:30,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 18:53:30,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 18:53:30,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 18:53:30,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 18:53:30,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-19 18:53:30,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-19 18:53:30,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-19 18:53:30,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-19 18:53:30,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-19 18:53:30,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-19 18:53:30,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-19 18:53:30,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-19 18:53:30,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-19 18:53:30,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-19 18:53:30,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-19 18:53:30,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-19 18:53:30,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-19 18:53:30,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-19 18:53:30,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-19 18:53:30,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-19 18:53:30,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-19 18:53:30,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-19 18:53:30,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-19 18:53:30,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-19 18:53:30,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-19 18:53:30,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-19 18:53:30,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-19 18:53:30,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-19 18:53:30,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-19 18:53:30,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-19 18:53:30,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-19 18:53:30,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-19 18:53:30,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-19 18:53:30,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-19 18:53:30,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-19 18:53:30,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-19 18:53:30,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-19 18:53:30,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-19 18:53:30,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-19 18:53:30,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-19 18:53:30,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-19 18:53:30,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-19 18:53:30,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-19 18:53:30,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-19 18:53:30,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-19 18:53:30,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-19 18:53:30,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-19 18:53:30,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-19 18:53:30,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-19 18:53:30,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-19 18:53:30,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-19 18:53:30,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-19 18:53:30,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-19 18:53:30,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-19 18:53:30,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-19 18:53:30,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-19 18:53:30,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-19 18:53:30,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-19 18:53:30,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-19 18:53:30,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-19 18:53:30,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-19 18:53:30,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-19 18:53:30,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-19 18:53:30,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-19 18:53:30,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-19 18:53:30,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-19 18:53:30,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-19 18:53:30,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-19 18:53:30,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-19 18:53:30,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-19 18:53:30,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-19 18:53:30,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-19 18:53:30,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-19 18:53:30,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-19 18:53:30,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-19 18:53:30,586 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002920572180767871); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.833333333333334E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 18:53:30,586 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-10-19 18:53:30,586 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.806811790177016E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002920572180767871); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.833333333333334E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-10-19 18:53:30,841 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 355 ms to try 5000 different iterations. Found a solution that moves 49 regions; Going from a computed imbalance of 6.806811790177016E-5 to a new imbalance of 8.8737528963636E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0054444444444444445); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 18:53:30,842 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1195393379:83 , srv2043419130:83 , srv2077732308:83 , srv2093823164:83 , srv210983931:83 , srv223530145:83 , srv358112045:83 , srv771600996:83 , srv824547989:83 , srv889895391:83 , srv1017359666:84 , srv1031917979:84 , srv1074541621:84 , srv1099175580:84 , srv1111521139:84 , srv1141125411:84 , srv115022089:84 , srv1159388076:84 , srv1173127194:84 , srv1181639057:84 , srv1189134801:84 , srv1238700620:84 , srv1241989146:84 , srv1283382444:84 , srv1303809273:84 , srv1313568469:84 , srv1344432469:84 , srv1350517833:84 , srv1363600636:84 , srv1376921598:84 , srv1379235248:84 , srv1413689296:84 , srv1440265411:84 , srv146240685:84 , srv1477071856:84 , srv1516805281:84 , srv1543910622:84 , srv1562672268:84 , srv159938100:84 , srv1617235643:84 , srv1680307410:84 , srv1695943230:84 , srv1707700838:84 , srv1748314348:84 , srv1748578283:84 , srv1797830770:84 , srv1849798095:84 , srv1870359988:84 , srv1875302350:84 , srv1909533388:84 , srv1935056484:84 , srv1947279920:84 , srv2024177023:84 , srv2036420180:84 , srv2043465693:84 , srv2052481948:84 , srv2087594516:84 , srv2095770785:84 , srv2138017684:84 , srv257950480:84 , srv267198752:84 , srv300081550:84 , srv304676882:84 , srv331349209:84 , srv355508678:84 , srv396621168:84 , srv45609493:84 , srv465208772:84 , srv483266234:84 , srv51961111:84 , srv567836301:84 , srv595812837:84 , srv617569491:84 , srv66271521:84 , srv664716809:84 , srv696773079:84 , srv735272239:84 , srv76309603:84 , srv841122059:84 , srv855347983:84 , srv871571782:84 , srv875179792:84 , srv878487320:84 , srv9278997:84 , srv930856481:84 , srv966133389:84 , srv969532819:84 , srv982238849:84 , srv996920502:84 , srv1066424162:85 , srv1098097628:85 , srv1309637023:85 , srv152443299:85 , srv1683316709:85 , srv1697736993:85 , srv2057246221:85 , srv399652278:85 , srv62831935:85 , srv73879462:85 , srv786483626:684 } 2023-10-19 18:53:30,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 18:53:30,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1875302350=45, srv1173127194=10, srv1617235643=34, srv1562672268=32, srv595812837=76, srv567836301=75, srv1413689296=25, srv1159388076=9, srv1017359666=0, srv1283382444=16, srv2077732308=55, srv875179792=91, srv1935056484=47, srv1309637023=18, srv930856481=95, srv1909533388=46, srv1748578283=41, srv1181639057=11, srv966133389=96, srv1683316709=36, srv399652278=70, srv159938100=33, srv2057246221=54, srv1543910622=31, srv2024177023=49, srv2036420180=50, srv855347983=89, srv735272239=82, srv2043465693=52, srv1350517833=21, srv210983931=59, srv62831935=78, srv2093823164=57, srv465208772=72, srv1313568469=19, srv996920502=99, srv617569491=77, srv664716809=80, srv841122059=88, srv1099175580=5, srv358112045=68, srv1241989146=15, srv257950480=62, srv1066424162=2, srv1344432469=20, srv2138017684=60, srv223530145=61, srv1074541621=3, srv300081550=64, srv1141125411=7, srv1195393379=13, srv73879462=83, srv1707700838=39, srv1238700620=14, srv267198752=63, srv2052481948=53, srv51961111=74, srv1947279920=48, srv1748314348=40, srv2043419130=51, srv1849798095=43, srv1516805281=29, srv771600996=85, srv66271521=79, srv1797830770=42, srv1363600636=22, srv1189134801=12, srv1379235248=24, srv355508678=67, srv45609493=71, srv331349209=66, srv1440265411=26, srv1303809273=17, srv9278997=94, srv696773079=81, srv871571782=90, srv396621168=69, srv1098097628=4, srv146240685=27, srv1376921598=23, srv2087594516=56, srv1697736993=38, srv483266234=73, srv982238849=98, srv1031917979=1, srv889895391=93, srv76309603=84, srv2095770785=58, srv115022089=8, srv969532819=97, srv1111521139=6, srv786483626=86, srv152443299=30, srv1695943230=37, srv1870359988=44, srv824547989=87, srv1477071856=28, srv304676882=65, srv1680307410=35, srv878487320=92} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 18:53:30,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 18:53:30,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 18:53:30,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 18:53:30,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 18:53:30,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 18:53:30,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 18:53:30,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 18:53:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 18:53:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 18:53:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 18:53:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 18:53:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 18:53:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 18:53:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 18:53:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 18:53:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 18:53:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 18:53:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 18:53:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 18:53:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 18:53:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 18:53:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 18:53:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 18:53:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 18:53:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 18:53:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 18:53:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 18:53:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 18:53:30,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 18:53:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 18:53:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-10-19 18:53:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-10-19 18:53:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-10-19 18:53:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-10-19 18:53:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-10-19 18:53:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-10-19 18:53:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-10-19 18:53:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-10-19 18:53:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-10-19 18:53:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-10-19 18:53:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-10-19 18:53:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-10-19 18:53:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-10-19 18:53:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-10-19 18:53:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-10-19 18:53:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-10-19 18:53:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-10-19 18:53:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-10-19 18:53:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-10-19 18:53:30,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-10-19 18:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-10-19 18:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-10-19 18:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-10-19 18:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-10-19 18:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-10-19 18:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-10-19 18:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-10-19 18:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-10-19 18:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-10-19 18:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-10-19 18:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-10-19 18:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-10-19 18:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-10-19 18:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-10-19 18:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-10-19 18:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-10-19 18:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-10-19 18:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-10-19 18:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-10-19 18:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-10-19 18:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-10-19 18:53:30,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-10-19 18:53:30,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-10-19 18:53:30,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-10-19 18:53:30,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-10-19 18:53:30,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-10-19 18:53:30,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-10-19 18:53:30,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-10-19 18:53:30,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-10-19 18:53:30,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-10-19 18:53:30,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-10-19 18:53:30,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-10-19 18:53:30,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-10-19 18:53:30,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-10-19 18:53:30,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-10-19 18:53:30,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-10-19 18:53:30,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-10-19 18:53:30,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-10-19 18:53:30,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-10-19 18:53:30,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-10-19 18:53:30,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-10-19 18:53:30,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-10-19 18:53:30,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-10-19 18:53:30,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-10-19 18:53:30,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-10-19 18:53:30,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-10-19 18:53:30,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-10-19 18:53:30,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-10-19 18:53:30,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-10-19 18:53:30,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-10-19 18:53:30,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-10-19 18:53:30,975 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=8.530647563026267E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 18:53:30,975 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1195393379:83 , srv2043419130:83 , srv2077732308:83 , srv2093823164:83 , srv210983931:83 , srv223530145:83 , srv358112045:83 , srv771600996:83 , srv824547989:83 , srv889895391:83 , srv1017359666:84 , srv1031917979:84 , srv1074541621:84 , srv1099175580:84 , srv1111521139:84 , srv1141125411:84 , srv115022089:84 , srv1159388076:84 , srv1173127194:84 , srv1181639057:84 , srv1189134801:84 , srv1238700620:84 , srv1241989146:84 , srv1283382444:84 , srv1303809273:84 , srv1313568469:84 , srv1344432469:84 , srv1350517833:84 , srv1363600636:84 , srv1376921598:84 , srv1379235248:84 , srv1413689296:84 , srv1440265411:84 , srv146240685:84 , srv1477071856:84 , srv1516805281:84 , srv1543910622:84 , srv1562672268:84 , srv159938100:84 , srv1617235643:84 , srv1680307410:84 , srv1695943230:84 , srv1707700838:84 , srv1748314348:84 , srv1748578283:84 , srv1797830770:84 , srv1849798095:84 , srv1870359988:84 , srv1875302350:84 , srv1909533388:84 , srv1935056484:84 , srv1947279920:84 , srv2024177023:84 , srv2036420180:84 , srv2043465693:84 , srv2052481948:84 , srv2087594516:84 , srv2095770785:84 , srv2138017684:84 , srv257950480:84 , srv267198752:84 , srv300081550:84 , srv304676882:84 , srv331349209:84 , srv355508678:84 , srv396621168:84 , srv45609493:84 , srv465208772:84 , srv483266234:84 , srv51961111:84 , srv567836301:84 , srv595812837:84 , srv617569491:84 , srv66271521:84 , srv664716809:84 , srv696773079:84 , srv735272239:84 , srv76309603:84 , srv841122059:84 , srv855347983:84 , srv871571782:84 , srv875179792:84 , srv878487320:84 , srv9278997:84 , srv930856481:84 , srv966133389:84 , srv969532819:84 , srv982238849:84 , srv996920502:84 , srv1066424162:85 , srv1098097628:85 , srv1309637023:85 , srv152443299:85 , srv1683316709:85 , srv1697736993:85 , srv2057246221:85 , srv399652278:85 , srv62831935:85 , srv73879462:85 , srv786483626:684 } 2023-10-19 18:53:31,002 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=317 (was 310) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=10757 (was 11264) 2023-10-19 18:53:31,013 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=317, ProcessCount=169, AvailableMemoryMB=10756 2023-10-19 18:53:31,013 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-10-19 18:53:31,013 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-10-19 18:53:31,022 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1082140123:84 , srv1268671290:84 , srv1335797548:84 , srv1381994989:84 , srv1395784832:84 , srv1521783853:84 , srv1586398305:84 , srv1604090748:84 , srv1634052564:84 , srv1758164840:84 , srv1771685044:84 , srv177894950:84 , srv1793856116:84 , srv1816797025:84 , srv1873080430:84 , srv1921407356:84 , srv2021295986:84 , srv2023058896:84 , srv212270741:84 , srv212451256:84 , srv299731539:84 , srv438995674:84 , srv516972498:84 , srv563830382:84 , srv595866561:84 , srv610727730:84 , srv678377180:84 , srv686267038:84 , srv820839253:84 , srv780250693:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-10-19 18:53:31,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 18:53:31,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv820839253=29, srv780250693=28, srv1604090748=7, srv1586398305=6, srv2023058896=17, srv1335797548=2, srv1395784832=4, srv1521783853=5, srv212451256=19, srv1873080430=14, srv212270741=18, srv1381994989=3, srv177894950=11, srv299731539=20, srv1921407356=15, srv678377180=26, srv1634052564=8, srv563830382=23, srv595866561=24, srv1758164840=9, srv1816797025=13, srv1793856116=12, srv2021295986=16, srv516972498=22, srv1082140123=0, srv686267038=27, srv1268671290=1, srv1771685044=10, srv438995674=21, srv610727730=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 18:53:31,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 18:53:31,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 18:53:31,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 18:53:31,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 18:53:31,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 18:53:31,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 18:53:31,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 18:53:31,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 18:53:31,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 18:53:31,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 18:53:31,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 18:53:31,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 18:53:31,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 18:53:31,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 18:53:31,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 18:53:31,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 18:53:31,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 18:53:31,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 18:53:31,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 18:53:31,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 18:53:31,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 18:53:31,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 18:53:31,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 18:53:31,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 18:53:31,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 18:53:31,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 18:53:31,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 18:53:31,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 18:53:31,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 18:53:31,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 18:53:31,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-19 18:53:31,050 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.106238586671064, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 18:53:31,050 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912527797581872, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.106238586671064, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-10-19 18:55:31,037 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5685341 different iterations. Found a solution that moves 1865 regions; Going from a computed imbalance of 0.9912527797581872 to a new imbalance of 4.353003038599517E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6907407407407408, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 18:55:31,044 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1082140123:90 , srv1268671290:90 , srv1335797548:90 , srv1381994989:90 , srv1395784832:90 , srv1521783853:90 , srv1586398305:90 , srv1604090748:90 , srv1634052564:90 , srv1758164840:90 , srv1771685044:90 , srv177894950:90 , srv1793856116:90 , srv1816797025:90 , srv1873080430:90 , srv1921407356:90 , srv2021295986:90 , srv2023058896:90 , srv212270741:90 , srv212451256:90 , srv299731539:90 , srv438995674:90 , srv516972498:90 , srv563830382:90 , srv595866561:90 , srv610727730:90 , srv678377180:90 , srv686267038:90 , srv780250693:90 , srv820839253:90 } 2023-10-19 18:55:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-10-19 18:55:31,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv820839253=29, srv780250693=28, srv1604090748=7, srv1586398305=6, srv2023058896=17, srv1335797548=2, srv1395784832=4, srv1521783853=5, srv212451256=19, srv1873080430=14, srv212270741=18, srv1381994989=3, srv177894950=11, srv299731539=20, srv1921407356=15, srv678377180=26, srv1634052564=8, srv563830382=23, srv595866561=24, srv1758164840=9, srv1816797025=13, srv1793856116=12, srv2021295986=16, srv516972498=22, srv1082140123=0, srv686267038=27, srv1268671290=1, srv1771685044=10, srv438995674=21, srv610727730=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-10-19 18:55:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-19 18:55:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-10-19 18:55:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-10-19 18:55:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-10-19 18:55:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-10-19 18:55:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-10-19 18:55:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-10-19 18:55:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-10-19 18:55:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-10-19 18:55:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-10-19 18:55:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-10-19 18:55:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-10-19 18:55:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-10-19 18:55:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-10-19 18:55:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-10-19 18:55:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-10-19 18:55:31,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-10-19 18:55:31,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-10-19 18:55:31,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-10-19 18:55:31,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-10-19 18:55:31,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-10-19 18:55:31,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-10-19 18:55:31,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-10-19 18:55:31,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-10-19 18:55:31,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-10-19 18:55:31,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-10-19 18:55:31,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-10-19 18:55:31,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-10-19 18:55:31,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-10-19 18:55:31,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-10-19 18:55:31,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-10-19 18:55:31,068 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-10-19 18:55:31,068 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1082140123:90 , srv1268671290:90 , srv1335797548:90 , srv1381994989:90 , srv1395784832:90 , srv1521783853:90 , srv1586398305:90 , srv1604090748:90 , srv1634052564:90 , srv1758164840:90 , srv1771685044:90 , srv177894950:90 , srv1793856116:90 , srv1816797025:90 , srv1873080430:90 , srv1921407356:90 , srv2021295986:90 , srv2023058896:90 , srv212270741:90 , srv212451256:90 , srv299731539:90 , srv438995674:90 , srv516972498:90 , srv563830382:90 , srv595866561:90 , srv610727730:90 , srv678377180:90 , srv686267038:90 , srv780250693:90 , srv820839253:90 } 2023-10-19 18:55:31,082 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=263 (was 317), ProcessCount=170 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=10535 (was 10756)