2023-05-30 06:53:22,373 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-05-30 06:53:22,657 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-05-30 06:53:22,756 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-05-30 06:53:22,811 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-30 06:53:22,818 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-30 06:53:22,831 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=296, ProcessCount=170, AvailableMemoryMB=4695 2023-05-30 06:53:22,833 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-30 06:53:22,833 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-30 06:53:22,965 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1013315937:84 , srv1015688271:84 , srv1049259908:84 , srv105985481:84 , srv1065950540:84 , srv1098554589:84 , srv1105804300:84 , srv1110024286:84 , srv1113585199:84 , srv1115501922:84 , srv1127904086:84 , srv1168214027:84 , srv1186392218:84 , srv1206652718:84 , srv121676787:84 , srv1224205280:84 , srv1242481169:84 , srv1248390958:84 , srv1279939135:84 , srv1313011017:84 , srv1337587997:84 , srv1341524478:84 , srv1353542818:84 , srv1359860626:84 , srv1420644952:84 , srv1421690070:84 , srv1423887042:84 , srv1439194559:84 , srv1466079977:84 , srv1495573040:84 , srv1511562450:84 , srv1529541612:84 , srv1548886412:84 , srv158276581:84 , srv1602028125:84 , srv1633959938:84 , srv1635652732:84 , srv1656927160:84 , srv1657780112:84 , srv166048081:84 , srv1680610345:84 , srv1691811241:84 , srv1731905308:84 , srv1753601754:84 , srv1754329277:84 , srv1759526055:84 , srv1763010278:84 , srv1769763198:84 , srv1798172470:84 , srv1807714441:84 , srv1823215167:84 , srv187703996:84 , srv1883780469:84 , srv2044580710:84 , srv20640568:84 , srv2077577913:84 , srv2087899047:84 , srv2100383264:84 , srv2114647104:84 , srv2123502182:84 , srv2141993649:84 , srv229781518:84 , srv238374417:84 , srv3062850:84 , srv324126524:84 , srv373126717:84 , srv405439271:84 , srv409842036:84 , srv432187852:84 , srv456289362:84 , srv460768953:84 , srv464557783:84 , srv466300415:84 , srv507757146:84 , srv571617097:84 , srv60080846:84 , srv623868796:84 , srv624290033:84 , srv669672934:84 , srv688868350:84 , srv698295683:84 , srv704067444:84 , srv732047733:84 , srv797144342:84 , srv808253903:84 , srv84369819:84 , srv850641707:84 , srv8574372:84 , srv869080211:84 , srv87707797:84 , srv882545536:84 , srv883704493:84 , srv901780905:84 , srv908847100:84 , srv918378873:84 , srv928129765:84 , srv957493237:84 , srv97718025:84 , srv996017812:84 , srv1392358657:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-05-30 06:53:22,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-30 06:53:22,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv869080211=89, srv1439194559=28, srv1495573040=30, srv1392358657=24, srv1763010278=47, srv1602028125=35, srv1127904086=10, srv918378873=95, srv1423887042=27, srv432187852=69, srv1691811241=42, srv1098554589=5, srv1420644952=25, srv121676787=14, srv1224205280=15, srv624290033=78, srv1657780112=39, srv105985481=3, srv1186392218=12, srv158276581=34, srv1754329277=45, srv669672934=79, srv87707797=90, srv1807714441=50, srv84369819=86, srv166048081=40, srv1548886412=33, srv1013315937=0, srv2123502182=60, srv1359860626=23, srv908847100=94, srv2141993649=61, srv1337587997=20, srv732047733=83, srv571617097=75, srv1680610345=41, srv928129765=96, srv704067444=82, srv1635652732=37, srv1353542818=22, srv60080846=76, srv1115501922=9, srv1065950540=4, srv1248390958=17, srv957493237=97, srv2077577913=56, srv456289362=70, srv623868796=77, srv97718025=98, srv324126524=65, srv373126717=66, srv901780905=93, srv1168214027=11, srv1049259908=2, srv2087899047=57, srv3062850=64, srv1798172470=49, srv1110024286=7, srv1823215167=51, srv507757146=74, srv1759526055=46, srv882545536=91, srv8574372=88, srv808253903=85, srv1421690070=26, srv2114647104=59, srv1529541612=32, srv1313011017=19, srv850641707=87, srv1511562450=31, srv1633959938=36, srv1341524478=21, srv20640568=55, srv797144342=84, srv1656927160=38, srv464557783=72, srv1015688271=1, srv1883780469=53, srv1466079977=29, srv1279939135=18, srv187703996=52, srv2044580710=54, srv2100383264=58, srv466300415=73, srv1242481169=16, srv688868350=80, srv698295683=81, srv883704493=92, srv996017812=99, srv409842036=68, srv460768953=71, srv1105804300=6, srv1731905308=43, srv1769763198=48, srv229781518=62, srv1206652718=13, srv238374417=63, srv1113585199=8, srv405439271=67, srv1753601754=44} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-30 06:53:23,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-30 06:53:23,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-30 06:53:23,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-30 06:53:23,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-30 06:53:23,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-30 06:53:23,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-30 06:53:23,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-30 06:53:23,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-30 06:53:23,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-30 06:53:23,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-30 06:53:23,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-30 06:53:23,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-30 06:53:23,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-30 06:53:23,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-30 06:53:23,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-30 06:53:23,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-30 06:53:23,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-30 06:53:23,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-30 06:53:23,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-30 06:53:23,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-30 06:53:23,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-30 06:53:23,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-30 06:53:23,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-30 06:53:23,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-30 06:53:23,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-30 06:53:23,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-30 06:53:23,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-30 06:53:23,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-30 06:53:23,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-30 06:53:23,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-30 06:53:23,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-30 06:53:23,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-30 06:53:23,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-30 06:53:23,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-30 06:53:23,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-30 06:53:23,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-30 06:53:23,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-30 06:53:23,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-30 06:53:23,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-30 06:53:23,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-30 06:53:23,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-30 06:53:23,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-30 06:53:23,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-30 06:53:23,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-30 06:53:23,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-30 06:53:23,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-30 06:53:23,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-30 06:53:23,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-30 06:53:23,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-30 06:53:23,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-30 06:53:23,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-30 06:53:23,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-30 06:53:23,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-30 06:53:23,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-30 06:53:23,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-30 06:53:23,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-30 06:53:23,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-30 06:53:23,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-30 06:53:23,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-30 06:53:23,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-30 06:53:23,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-30 06:53:23,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-30 06:53:23,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-30 06:53:23,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-30 06:53:23,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-30 06:53:23,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-30 06:53:23,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-30 06:53:23,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-30 06:53:23,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-30 06:53:23,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-30 06:53:23,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-30 06:53:23,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-30 06:53:23,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-30 06:53:23,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-30 06:53:23,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-30 06:53:23,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-30 06:53:23,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-30 06:53:23,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-30 06:53:23,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-30 06:53:23,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-30 06:53:23,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-30 06:53:23,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-30 06:53:23,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-30 06:53:23,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-30 06:53:23,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-30 06:53:23,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-30 06:53:23,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-30 06:53:23,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-30 06:53:23,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-30 06:53:23,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-30 06:53:23,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-30 06:53:23,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-30 06:53:23,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-30 06:53:23,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-30 06:53:23,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-30 06:53:23,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-30 06:53:23,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-30 06:53:23,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-30 06:53:23,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-30 06:53:23,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-30 06:53:23,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-30 06:53:23,259 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 06:53:23,259 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-30 06:53:23,259 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-30 06:53:23,533 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 434 ms to try 5000 different iterations. Found a solution that moves 4658 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1858053770557023. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03980310688401913, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05590596562213785, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5175555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03980310688401913, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18091666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24941666666666668, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 06:53:23,550 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv918378873:80 , srv996017812:80 , srv1353542818:81 , srv850641707:81 , srv1115501922:82 , srv1168214027:82 , srv1529541612:82 , srv238374417:82 , srv624290033:82 , srv704067444:82 , srv957493237:82 , srv97718025:82 , srv121676787:83 , srv1279939135:83 , srv1635652732:83 , srv1657780112:83 , srv2100383264:83 , srv405439271:83 , srv84369819:83 , srv8574372:83 , srv1127904086:84 , srv1633959938:84 , srv1807714441:84 , srv3062850:84 , srv456289362:84 , srv1098554589:85 , srv1105804300:85 , srv1359860626:85 , srv1421690070:85 , srv1466079977:85 , srv1511562450:85 , srv1656927160:85 , srv1680610345:85 , srv2044580710:85 , srv324126524:85 , srv60080846:85 , srv87707797:85 , srv1015688271:86 , srv1049259908:86 , srv1065950540:86 , srv1113585199:86 , srv1313011017:86 , srv1602028125:86 , srv1753601754:86 , srv1754329277:86 , srv187703996:86 , srv1883780469:86 , srv2087899047:86 , srv460768953:86 , srv571617097:86 , srv732047733:86 , srv1224205280:87 , srv1420644952:87 , srv1423887042:87 , srv158276581:87 , srv1759526055:87 , srv1769763198:87 , srv1823215167:87 , srv464557783:87 , srv466300415:87 , srv623868796:87 , srv669672934:87 , srv808253903:87 , srv869080211:87 , srv882545536:87 , srv908847100:87 , srv928129765:87 , srv1242481169:88 , srv1341524478:88 , srv1495573040:88 , srv1548886412:88 , srv1691811241:88 , srv1763010278:88 , srv20640568:88 , srv2114647104:88 , srv2123502182:88 , srv229781518:88 , srv507757146:88 , srv688868350:88 , srv883704493:88 , srv1013315937:89 , srv1248390958:89 , srv1337587997:89 , srv1439194559:89 , srv1731905308:89 , srv409842036:89 , srv698295683:89 , srv797144342:89 , srv1798172470:90 , srv2141993649:90 , srv373126717:90 , srv432187852:90 , srv105985481:91 , srv1206652718:91 , srv901780905:91 , srv1110024286:93 , srv1186392218:98 , srv166048081:98 , srv2077577913:101 , srv1392358657:443 } 2023-05-30 06:53:23,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-30 06:53:23,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv869080211=89, srv1439194559=28, srv1495573040=30, srv1392358657=24, srv1763010278=47, srv1602028125=35, srv1127904086=10, srv918378873=95, srv1423887042=27, srv432187852=69, srv1691811241=42, srv1098554589=5, srv1420644952=25, srv121676787=14, srv1224205280=15, srv624290033=78, srv1657780112=39, srv105985481=3, srv1186392218=12, srv158276581=34, srv1754329277=45, srv669672934=79, srv87707797=90, srv1807714441=50, srv84369819=86, srv166048081=40, srv1548886412=33, srv1013315937=0, srv2123502182=60, srv1359860626=23, srv908847100=94, srv2141993649=61, srv1337587997=20, srv732047733=83, srv571617097=75, srv1680610345=41, srv928129765=96, srv704067444=82, srv1635652732=37, srv1353542818=22, srv60080846=76, srv1115501922=9, srv1065950540=4, srv1248390958=17, srv957493237=97, srv2077577913=56, srv456289362=70, srv623868796=77, srv97718025=98, srv324126524=65, srv373126717=66, srv901780905=93, srv1168214027=11, srv1049259908=2, srv2087899047=57, srv3062850=64, srv1798172470=49, srv1110024286=7, srv1823215167=51, srv507757146=74, srv1759526055=46, srv882545536=91, srv8574372=88, srv808253903=85, srv1421690070=26, srv2114647104=59, srv1529541612=32, srv1313011017=19, srv850641707=87, srv1511562450=31, srv1633959938=36, srv1341524478=21, srv20640568=55, srv797144342=84, srv1656927160=38, srv464557783=72, srv1015688271=1, srv1883780469=53, srv1466079977=29, srv1279939135=18, srv187703996=52, srv2044580710=54, srv2100383264=58, srv466300415=73, srv1242481169=16, srv688868350=80, srv698295683=81, srv883704493=92, srv996017812=99, srv409842036=68, srv460768953=71, srv1105804300=6, srv1731905308=43, srv1769763198=48, srv229781518=62, srv1206652718=13, srv238374417=63, srv1113585199=8, srv405439271=67, srv1753601754=44} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-30 06:53:23,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-30 06:53:23,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-30 06:53:23,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-30 06:53:23,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-30 06:53:23,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-30 06:53:23,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-30 06:53:23,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-30 06:53:23,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-30 06:53:23,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-30 06:53:23,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-30 06:53:23,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-30 06:53:23,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-30 06:53:23,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-30 06:53:23,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-30 06:53:23,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-30 06:53:23,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-30 06:53:23,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-30 06:53:23,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-30 06:53:23,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-30 06:53:23,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-30 06:53:23,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-30 06:53:23,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-30 06:53:23,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-30 06:53:23,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-30 06:53:23,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-30 06:53:23,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-30 06:53:23,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-30 06:53:23,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-30 06:53:23,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-30 06:53:23,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-30 06:53:23,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-30 06:53:23,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-30 06:53:23,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-30 06:53:23,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-30 06:53:23,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-30 06:53:23,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-30 06:53:23,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-30 06:53:23,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-30 06:53:23,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-30 06:53:23,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-30 06:53:23,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-30 06:53:23,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-30 06:53:23,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-30 06:53:23,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-30 06:53:23,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-30 06:53:23,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-30 06:53:23,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-30 06:53:23,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-30 06:53:23,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-30 06:53:23,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-30 06:53:23,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-30 06:53:23,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-30 06:53:23,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-30 06:53:23,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-30 06:53:23,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-30 06:53:23,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-30 06:53:23,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-30 06:53:23,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-30 06:53:23,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-30 06:53:23,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-30 06:53:23,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-30 06:53:23,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-30 06:53:23,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-30 06:53:23,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-30 06:53:23,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-30 06:53:23,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-30 06:53:23,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-30 06:53:23,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-30 06:53:23,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-30 06:53:23,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-30 06:53:23,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-30 06:53:23,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-30 06:53:23,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-30 06:53:23,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-30 06:53:23,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-30 06:53:23,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-30 06:53:23,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-30 06:53:23,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-30 06:53:23,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-30 06:53:23,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-30 06:53:23,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-30 06:53:23,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-30 06:53:23,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-30 06:53:23,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-30 06:53:23,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-30 06:53:23,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-30 06:53:23,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-30 06:53:23,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-30 06:53:23,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-30 06:53:23,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-30 06:53:23,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-30 06:53:23,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-30 06:53:23,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-30 06:53:23,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-30 06:53:23,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-30 06:53:23,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-30 06:53:23,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-30 06:53:23,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-30 06:53:23,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-30 06:53:23,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-30 06:53:23,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-30 06:53:23,709 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03980310688401913, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05590596562213785, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03980310688401913, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18091666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24941666666666668, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 06:53:23,709 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-30 06:53:23,709 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18577276104258628, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03980310688401913, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05590596562213785, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03980310688401913, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18091666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24941666666666668, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-30 06:53:24,018 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 419 ms to try 5000 different iterations. Found a solution that moves 2699 regions; Going from a computed imbalance of 0.18577276104258628 to a new imbalance of 0.014819795698786466. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02127439951731416); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03531484174023787, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29988888888888887, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02127439951731416); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01175); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 06:53:24,031 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1186392218:74 , srv2077577913:74 , srv166048081:76 , srv1110024286:78 , srv1206652718:79 , srv1439194559:80 , srv901780905:80 , srv1337587997:81 , srv432187852:81 , srv1248390958:82 , srv1731905308:82 , srv1798172470:82 , srv2141993649:82 , srv373126717:82 , srv1013315937:83 , srv105985481:83 , srv1242481169:83 , srv20640568:83 , srv2114647104:83 , srv688868350:83 , srv928129765:83 , srv1113585199:84 , srv1548886412:84 , srv1763010278:84 , srv1769763198:84 , srv1823215167:84 , srv409842036:84 , srv466300415:84 , srv507757146:84 , srv882545536:84 , srv883704493:84 , srv1065950540:85 , srv1224205280:85 , srv1313011017:85 , srv1341524478:85 , srv1420644952:85 , srv158276581:85 , srv1656927160:85 , srv1680610345:85 , srv1759526055:85 , srv187703996:85 , srv2123502182:85 , srv229781518:85 , srv669672934:85 , srv732047733:85 , srv797144342:85 , srv808253903:85 , srv87707797:85 , srv1049259908:86 , srv1279939135:86 , srv1495573040:86 , srv1691811241:86 , srv1754329277:86 , srv1883780469:86 , srv2044580710:86 , srv2087899047:86 , srv324126524:86 , srv464557783:86 , srv698295683:86 , srv869080211:86 , srv1098554589:87 , srv1105804300:87 , srv1127904086:87 , srv1602028125:87 , srv1633959938:87 , srv1657780112:87 , srv1753601754:87 , srv460768953:87 , srv571617097:87 , srv908847100:87 , srv1168214027:88 , srv121676787:88 , srv1359860626:88 , srv1423887042:88 , srv1466079977:88 , srv456289362:88 , srv623868796:88 , srv1015688271:89 , srv1807714441:89 , srv238374417:89 , srv405439271:89 , srv704067444:89 , srv97718025:89 , srv1115501922:90 , srv1421690070:90 , srv1511562450:90 , srv1635652732:90 , srv2100383264:90 , srv3062850:90 , srv60080846:90 , srv624290033:90 , srv84369819:90 , srv918378873:90 , srv1353542818:91 , srv1529541612:91 , srv8574372:91 , srv996017812:91 , srv957493237:92 , srv850641707:93 , srv1392358657:520 } 2023-05-30 06:53:24,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-30 06:53:24,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv869080211=89, srv1439194559=28, srv1495573040=30, srv1392358657=24, srv1763010278=47, srv1602028125=35, srv1127904086=10, srv918378873=95, srv1423887042=27, srv432187852=69, srv1691811241=42, srv1098554589=5, srv1420644952=25, srv121676787=14, srv1224205280=15, srv624290033=78, srv1657780112=39, srv105985481=3, srv1186392218=12, srv158276581=34, srv1754329277=45, srv669672934=79, srv87707797=90, srv1807714441=50, srv84369819=86, srv166048081=40, srv1548886412=33, srv1013315937=0, srv2123502182=60, srv1359860626=23, srv908847100=94, srv2141993649=61, srv1337587997=20, srv732047733=83, srv571617097=75, srv1680610345=41, srv928129765=96, srv704067444=82, srv1635652732=37, srv1353542818=22, srv60080846=76, srv1115501922=9, srv1065950540=4, srv1248390958=17, srv957493237=97, srv2077577913=56, srv456289362=70, srv623868796=77, srv97718025=98, srv324126524=65, srv373126717=66, srv901780905=93, srv1168214027=11, srv1049259908=2, srv2087899047=57, srv3062850=64, srv1798172470=49, srv1110024286=7, srv1823215167=51, srv507757146=74, srv1759526055=46, srv882545536=91, srv8574372=88, srv808253903=85, srv1421690070=26, srv2114647104=59, srv1529541612=32, srv1313011017=19, srv850641707=87, srv1511562450=31, srv1633959938=36, srv1341524478=21, srv20640568=55, srv797144342=84, srv1656927160=38, srv464557783=72, srv1015688271=1, srv1883780469=53, srv1466079977=29, srv1279939135=18, srv187703996=52, srv2044580710=54, srv2100383264=58, srv466300415=73, srv1242481169=16, srv688868350=80, srv698295683=81, srv883704493=92, srv996017812=99, srv409842036=68, srv460768953=71, srv1105804300=6, srv1731905308=43, srv1769763198=48, srv229781518=62, srv1206652718=13, srv238374417=63, srv1113585199=8, srv405439271=67, srv1753601754=44} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-30 06:53:24,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-30 06:53:24,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-30 06:53:24,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-30 06:53:24,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-30 06:53:24,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-30 06:53:24,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-30 06:53:24,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-30 06:53:24,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-30 06:53:24,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-30 06:53:24,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-30 06:53:24,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-30 06:53:24,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-30 06:53:24,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-30 06:53:24,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-30 06:53:24,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-30 06:53:24,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-30 06:53:24,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-30 06:53:24,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-30 06:53:24,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-30 06:53:24,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-30 06:53:24,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-30 06:53:24,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-30 06:53:24,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-30 06:53:24,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-30 06:53:24,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-30 06:53:24,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-30 06:53:24,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-30 06:53:24,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-30 06:53:24,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-30 06:53:24,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-30 06:53:24,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-30 06:53:24,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-30 06:53:24,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-30 06:53:24,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-30 06:53:24,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-30 06:53:24,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-30 06:53:24,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-30 06:53:24,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-30 06:53:24,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-30 06:53:24,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-30 06:53:24,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-30 06:53:24,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-30 06:53:24,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-30 06:53:24,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-30 06:53:24,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-30 06:53:24,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-30 06:53:24,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-30 06:53:24,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-30 06:53:24,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-30 06:53:24,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-30 06:53:24,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-30 06:53:24,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-30 06:53:24,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-30 06:53:24,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-30 06:53:24,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-30 06:53:24,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-30 06:53:24,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-30 06:53:24,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-30 06:53:24,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-30 06:53:24,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-30 06:53:24,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-30 06:53:24,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-30 06:53:24,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-30 06:53:24,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-30 06:53:24,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-30 06:53:24,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-30 06:53:24,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-30 06:53:24,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-30 06:53:24,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-30 06:53:24,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-30 06:53:24,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-30 06:53:24,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-30 06:53:24,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-30 06:53:24,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-30 06:53:24,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-30 06:53:24,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-30 06:53:24,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-30 06:53:24,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-30 06:53:24,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-30 06:53:24,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-30 06:53:24,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-30 06:53:24,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-30 06:53:24,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-30 06:53:24,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-30 06:53:24,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-30 06:53:24,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-30 06:53:24,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-30 06:53:24,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-30 06:53:24,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-30 06:53:24,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-30 06:53:24,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-30 06:53:24,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-30 06:53:24,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-30 06:53:24,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-30 06:53:24,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-30 06:53:24,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-30 06:53:24,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-30 06:53:24,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-30 06:53:24,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-30 06:53:24,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-30 06:53:24,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-30 06:53:24,190 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02127439951731416); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03531484174023787, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02127439951731416); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01175); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 06:53:24,190 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-30 06:53:24,190 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014800896896854272, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02127439951731416); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03531484174023787, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02127439951731416); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01175); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-30 06:53:24,507 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 433 ms to try 5000 different iterations. Found a solution that moves 1013 regions; Going from a computed imbalance of 0.014800896896854272 to a new imbalance of 9.368657985100995E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006108304630544682); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010016820869752056); MoveCostFunction : (multiplier=7.0, imbalance=0.11255555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006108304630544682); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0045); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 06:53:24,511 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv850641707:82 , srv1421690070:83 , srv1423887042:83 , srv1511562450:83 , srv166048081:83 , srv1691811241:83 , srv2077577913:83 , srv60080846:83 , srv623868796:83 , srv698295683:83 , srv8574372:83 , srv1015688271:84 , srv1341524478:84 , srv1359860626:84 , srv1466079977:84 , srv1495573040:84 , srv1529541612:84 , srv1753601754:84 , srv1807714441:84 , srv3062850:84 , srv409842036:84 , srv460768953:84 , srv464557783:84 , srv869080211:84 , srv908847100:84 , srv957493237:84 , srv1013315937:85 , srv1049259908:85 , srv1098554589:85 , srv1186392218:85 , srv1224205280:85 , srv1313011017:85 , srv1353542818:85 , srv1420644952:85 , srv1602028125:85 , srv1635652732:85 , srv1759526055:85 , srv2044580710:85 , srv2100383264:85 , srv2123502182:85 , srv229781518:85 , srv405439271:85 , srv507757146:85 , srv571617097:85 , srv624290033:85 , srv797144342:85 , srv808253903:85 , srv84369819:85 , srv883704493:85 , srv105985481:86 , srv1065950540:86 , srv1110024286:86 , srv1115501922:86 , srv1127904086:86 , srv121676787:86 , srv1548886412:86 , srv158276581:86 , srv1633959938:86 , srv1731905308:86 , srv1754329277:86 , srv1763010278:86 , srv1769763198:86 , srv1798172470:86 , srv1823215167:86 , srv187703996:86 , srv1883780469:86 , srv20640568:86 , srv2087899047:86 , srv2114647104:86 , srv2141993649:86 , srv238374417:86 , srv373126717:86 , srv456289362:86 , srv466300415:86 , srv669672934:86 , srv704067444:86 , srv732047733:86 , srv882545536:86 , srv901780905:86 , srv996017812:86 , srv1105804300:87 , srv1113585199:87 , srv1168214027:87 , srv1206652718:87 , srv1242481169:87 , srv1248390958:87 , srv1439194559:87 , srv1656927160:87 , srv1657780112:87 , srv324126524:87 , srv432187852:87 , srv688868350:87 , srv87707797:87 , srv918378873:87 , srv97718025:87 , srv1279939135:88 , srv1337587997:88 , srv928129765:88 , srv1680610345:89 , srv1392358657:549 } 2023-05-30 06:53:24,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-30 06:53:24,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv869080211=89, srv1439194559=28, srv1495573040=30, srv1392358657=24, srv1763010278=47, srv1602028125=35, srv1127904086=10, srv918378873=95, srv1423887042=27, srv432187852=69, srv1691811241=42, srv1098554589=5, srv1420644952=25, srv121676787=14, srv1224205280=15, srv624290033=78, srv1657780112=39, srv105985481=3, srv1186392218=12, srv158276581=34, srv1754329277=45, srv669672934=79, srv87707797=90, srv1807714441=50, srv84369819=86, srv166048081=40, srv1548886412=33, srv1013315937=0, srv2123502182=60, srv1359860626=23, srv908847100=94, srv2141993649=61, srv1337587997=20, srv732047733=83, srv571617097=75, srv1680610345=41, srv928129765=96, srv704067444=82, srv1635652732=37, srv1353542818=22, srv60080846=76, srv1115501922=9, srv1065950540=4, srv1248390958=17, srv957493237=97, srv2077577913=56, srv456289362=70, srv623868796=77, srv97718025=98, srv324126524=65, srv373126717=66, srv901780905=93, srv1168214027=11, srv1049259908=2, srv2087899047=57, srv3062850=64, srv1798172470=49, srv1110024286=7, srv1823215167=51, srv507757146=74, srv1759526055=46, srv882545536=91, srv8574372=88, srv808253903=85, srv1421690070=26, srv2114647104=59, srv1529541612=32, srv1313011017=19, srv850641707=87, srv1511562450=31, srv1633959938=36, srv1341524478=21, srv20640568=55, srv797144342=84, srv1656927160=38, srv464557783=72, srv1015688271=1, srv1883780469=53, srv1466079977=29, srv1279939135=18, srv187703996=52, srv2044580710=54, srv2100383264=58, srv466300415=73, srv1242481169=16, srv688868350=80, srv698295683=81, srv883704493=92, srv996017812=99, srv409842036=68, srv460768953=71, srv1105804300=6, srv1731905308=43, srv1769763198=48, srv229781518=62, srv1206652718=13, srv238374417=63, srv1113585199=8, srv405439271=67, srv1753601754=44} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-30 06:53:24,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-30 06:53:24,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-30 06:53:24,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-30 06:53:24,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-30 06:53:24,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-30 06:53:24,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-30 06:53:24,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-30 06:53:24,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-30 06:53:24,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-30 06:53:24,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-30 06:53:24,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-30 06:53:24,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-30 06:53:24,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-30 06:53:24,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-30 06:53:24,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-30 06:53:24,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-30 06:53:24,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-30 06:53:24,539 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-30 06:53:24,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-30 06:53:24,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-30 06:53:24,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-30 06:53:24,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-30 06:53:24,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-30 06:53:24,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-30 06:53:24,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-30 06:53:24,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-30 06:53:24,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-30 06:53:24,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-30 06:53:24,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-30 06:53:24,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-30 06:53:24,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-30 06:53:24,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-30 06:53:24,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-30 06:53:24,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-30 06:53:24,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-30 06:53:24,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-30 06:53:24,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-30 06:53:24,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-30 06:53:24,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-30 06:53:24,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-30 06:53:24,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-30 06:53:24,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-30 06:53:24,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-30 06:53:24,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-30 06:53:24,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-30 06:53:24,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-30 06:53:24,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-30 06:53:24,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-30 06:53:24,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-30 06:53:24,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-30 06:53:24,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-30 06:53:24,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-30 06:53:24,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-30 06:53:24,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-30 06:53:24,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-30 06:53:24,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-30 06:53:24,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-30 06:53:24,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-30 06:53:24,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-30 06:53:24,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-30 06:53:24,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-30 06:53:24,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-30 06:53:24,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-30 06:53:24,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-30 06:53:24,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-30 06:53:24,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-30 06:53:24,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-30 06:53:24,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-30 06:53:24,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-30 06:53:24,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-30 06:53:24,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-30 06:53:24,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-30 06:53:24,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-30 06:53:24,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-30 06:53:24,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-30 06:53:24,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-30 06:53:24,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-30 06:53:24,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-30 06:53:24,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-30 06:53:24,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-30 06:53:24,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-30 06:53:24,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-30 06:53:24,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-30 06:53:24,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-30 06:53:24,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-30 06:53:24,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-30 06:53:24,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-30 06:53:24,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-30 06:53:24,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-30 06:53:24,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-30 06:53:24,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-30 06:53:24,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-30 06:53:24,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-30 06:53:24,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-30 06:53:24,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-30 06:53:24,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-30 06:53:24,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-30 06:53:24,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-30 06:53:24,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-30 06:53:24,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-30 06:53:24,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-30 06:53:24,676 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006108304630544682); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010016820869752056); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006108304630544682); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0045); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 06:53:24,676 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-30 06:53:24,677 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.297726209045745E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006108304630544682); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010016820869752056); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006108304630544682); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0045); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-30 06:53:24,934 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 365 ms to try 5000 different iterations. Found a solution that moves 285 regions; Going from a computed imbalance of 9.297726209045745E-4 to a new imbalance of 3.7101830543973585E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023210354127426377); MoveCostFunction : (multiplier=7.0, imbalance=0.03166666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 06:53:24,935 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1015688271:83 , srv105985481:83 , srv1105804300:83 , srv1680610345:83 , srv797144342:83 , srv1013315937:84 , srv1115501922:84 , srv1242481169:84 , srv1248390958:84 , srv1337587997:84 , srv1421690070:84 , srv1495573040:84 , srv1511562450:84 , srv1548886412:84 , srv158276581:84 , srv1602028125:84 , srv1635652732:84 , srv1657780112:84 , srv1754329277:84 , srv1763010278:84 , srv1798172470:84 , srv1883780469:84 , srv2077577913:84 , srv2100383264:84 , srv2123502182:84 , srv229781518:84 , srv238374417:84 , srv3062850:84 , srv324126524:84 , srv373126717:84 , srv432187852:84 , srv456289362:84 , srv464557783:84 , srv571617097:84 , srv60080846:84 , srv669672934:84 , srv688868350:84 , srv698295683:84 , srv732047733:84 , srv84369819:84 , srv883704493:84 , srv908847100:84 , srv928129765:84 , srv957493237:84 , srv97718025:84 , srv1049259908:85 , srv1065950540:85 , srv1098554589:85 , srv1110024286:85 , srv1113585199:85 , srv1127904086:85 , srv1168214027:85 , srv1186392218:85 , srv1206652718:85 , srv121676787:85 , srv1224205280:85 , srv1279939135:85 , srv1313011017:85 , srv1341524478:85 , srv1353542818:85 , srv1359860626:85 , srv1423887042:85 , srv1466079977:85 , srv1529541612:85 , srv1633959938:85 , srv1656927160:85 , srv166048081:85 , srv1691811241:85 , srv1731905308:85 , srv1753601754:85 , srv1759526055:85 , srv1769763198:85 , srv1823215167:85 , srv187703996:85 , srv20640568:85 , srv2087899047:85 , srv2141993649:85 , srv405439271:85 , srv409842036:85 , srv460768953:85 , srv466300415:85 , srv507757146:85 , srv623868796:85 , srv624290033:85 , srv704067444:85 , srv808253903:85 , srv8574372:85 , srv869080211:85 , srv87707797:85 , srv882545536:85 , srv901780905:85 , srv918378873:85 , srv996017812:85 , srv1420644952:86 , srv1439194559:86 , srv1807714441:86 , srv2044580710:86 , srv2114647104:86 , srv850641707:86 , srv1392358657:629 } 2023-05-30 06:53:24,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-30 06:53:24,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv869080211=89, srv1439194559=28, srv1495573040=30, srv1392358657=24, srv1763010278=47, srv1602028125=35, srv1127904086=10, srv918378873=95, srv1423887042=27, srv432187852=69, srv1691811241=42, srv1098554589=5, srv1420644952=25, srv121676787=14, srv1224205280=15, srv624290033=78, srv1657780112=39, srv105985481=3, srv1186392218=12, srv158276581=34, srv1754329277=45, srv669672934=79, srv87707797=90, srv1807714441=50, srv84369819=86, srv166048081=40, srv1548886412=33, srv1013315937=0, srv2123502182=60, srv1359860626=23, srv908847100=94, srv2141993649=61, srv1337587997=20, srv732047733=83, srv571617097=75, srv1680610345=41, srv928129765=96, srv704067444=82, srv1635652732=37, srv1353542818=22, srv60080846=76, srv1115501922=9, srv1065950540=4, srv1248390958=17, srv957493237=97, srv2077577913=56, srv456289362=70, srv623868796=77, srv97718025=98, srv324126524=65, srv373126717=66, srv901780905=93, srv1168214027=11, srv1049259908=2, srv2087899047=57, srv3062850=64, srv1798172470=49, srv1110024286=7, srv1823215167=51, srv507757146=74, srv1759526055=46, srv882545536=91, srv8574372=88, srv808253903=85, srv1421690070=26, srv2114647104=59, srv1529541612=32, srv1313011017=19, srv850641707=87, srv1511562450=31, srv1633959938=36, srv1341524478=21, srv20640568=55, srv797144342=84, srv1656927160=38, srv464557783=72, srv1015688271=1, srv1883780469=53, srv1466079977=29, srv1279939135=18, srv187703996=52, srv2044580710=54, srv2100383264=58, srv466300415=73, srv1242481169=16, srv688868350=80, srv698295683=81, srv883704493=92, srv996017812=99, srv409842036=68, srv460768953=71, srv1105804300=6, srv1731905308=43, srv1769763198=48, srv229781518=62, srv1206652718=13, srv238374417=63, srv1113585199=8, srv405439271=67, srv1753601754=44} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-30 06:53:24,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-30 06:53:24,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-30 06:53:24,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-30 06:53:24,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-30 06:53:24,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-30 06:53:24,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-30 06:53:24,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-30 06:53:24,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-30 06:53:24,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-30 06:53:24,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-30 06:53:24,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-30 06:53:24,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-30 06:53:24,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-30 06:53:24,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-30 06:53:24,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-30 06:53:24,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-30 06:53:24,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-30 06:53:24,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-30 06:53:24,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-30 06:53:24,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-30 06:53:24,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-30 06:53:24,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-30 06:53:24,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-30 06:53:24,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-30 06:53:24,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-30 06:53:24,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-30 06:53:24,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-30 06:53:24,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-30 06:53:24,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-30 06:53:24,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-30 06:53:24,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-30 06:53:24,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-30 06:53:24,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-30 06:53:24,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-30 06:53:24,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-30 06:53:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-30 06:53:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-30 06:53:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-30 06:53:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-30 06:53:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-30 06:53:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-30 06:53:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-30 06:53:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-30 06:53:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-30 06:53:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-30 06:53:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-30 06:53:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-30 06:53:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-30 06:53:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-30 06:53:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-30 06:53:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-30 06:53:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-30 06:53:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-30 06:53:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-30 06:53:24,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-30 06:53:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-30 06:53:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-30 06:53:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-30 06:53:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-30 06:53:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-30 06:53:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-30 06:53:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-30 06:53:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-30 06:53:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-30 06:53:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-30 06:53:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-30 06:53:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-30 06:53:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-30 06:53:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-30 06:53:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-30 06:53:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-30 06:53:24,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-30 06:53:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-30 06:53:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-30 06:53:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-30 06:53:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-30 06:53:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-30 06:53:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-30 06:53:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-30 06:53:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-30 06:53:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-30 06:53:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-30 06:53:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-30 06:53:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-30 06:53:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-30 06:53:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-30 06:53:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-30 06:53:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-30 06:53:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-30 06:53:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-30 06:53:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-30 06:53:24,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-30 06:53:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-30 06:53:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-30 06:53:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-30 06:53:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-30 06:53:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-30 06:53:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-30 06:53:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-30 06:53:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-30 06:53:24,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-30 06:53:25,059 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023210354127426377); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 06:53:25,059 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-30 06:53:25,059 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.510621789088909E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023210354127426377); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-30 06:53:25,281 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 316 ms to try 5000 different iterations. Found a solution that moves 33 regions; Going from a computed imbalance of 3.510621789088909E-5 to a new imbalance of 7.920492739128986E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0036666666666666666); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 06:53:25,281 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv105985481:83 , srv1420644952:83 , srv1423887042:83 , srv1807714441:83 , srv2087899047:83 , srv2114647104:83 , srv2141993649:83 , srv623868796:83 , srv1015688271:84 , srv1049259908:84 , srv1065950540:84 , srv1098554589:84 , srv1105804300:84 , srv1110024286:84 , srv1113585199:84 , srv1115501922:84 , srv1127904086:84 , srv1168214027:84 , srv1186392218:84 , srv1206652718:84 , srv121676787:84 , srv1224205280:84 , srv1242481169:84 , srv1248390958:84 , srv1279939135:84 , srv1337587997:84 , srv1341524478:84 , srv1353542818:84 , srv1359860626:84 , srv1421690070:84 , srv1439194559:84 , srv1466079977:84 , srv1495573040:84 , srv1511562450:84 , srv1529541612:84 , srv1548886412:84 , srv158276581:84 , srv1602028125:84 , srv1633959938:84 , srv1635652732:84 , srv1656927160:84 , srv166048081:84 , srv1680610345:84 , srv1691811241:84 , srv1731905308:84 , srv1753601754:84 , srv1754329277:84 , srv1759526055:84 , srv1763010278:84 , srv1769763198:84 , srv1798172470:84 , srv1823215167:84 , srv187703996:84 , srv1883780469:84 , srv2044580710:84 , srv20640568:84 , srv2077577913:84 , srv2100383264:84 , srv2123502182:84 , srv229781518:84 , srv3062850:84 , srv324126524:84 , srv373126717:84 , srv405439271:84 , srv409842036:84 , srv432187852:84 , srv456289362:84 , srv460768953:84 , srv466300415:84 , srv507757146:84 , srv571617097:84 , srv60080846:84 , srv624290033:84 , srv669672934:84 , srv688868350:84 , srv698295683:84 , srv704067444:84 , srv797144342:84 , srv808253903:84 , srv84369819:84 , srv850641707:84 , srv8574372:84 , srv869080211:84 , srv87707797:84 , srv882545536:84 , srv901780905:84 , srv908847100:84 , srv918378873:84 , srv928129765:84 , srv957493237:84 , srv97718025:84 , srv996017812:84 , srv1013315937:85 , srv1313011017:85 , srv1657780112:85 , srv238374417:85 , srv464557783:85 , srv732047733:85 , srv883704493:85 , srv1392358657:685 } 2023-05-30 06:53:25,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-30 06:53:25,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv869080211=89, srv1439194559=28, srv1495573040=30, srv1392358657=24, srv1763010278=47, srv1602028125=35, srv1127904086=10, srv918378873=95, srv1423887042=27, srv432187852=69, srv1691811241=42, srv1098554589=5, srv1420644952=25, srv121676787=14, srv1224205280=15, srv624290033=78, srv1657780112=39, srv105985481=3, srv1186392218=12, srv158276581=34, srv1754329277=45, srv669672934=79, srv87707797=90, srv1807714441=50, srv84369819=86, srv166048081=40, srv1548886412=33, srv1013315937=0, srv2123502182=60, srv1359860626=23, srv908847100=94, srv2141993649=61, srv1337587997=20, srv732047733=83, srv571617097=75, srv1680610345=41, srv928129765=96, srv704067444=82, srv1635652732=37, srv1353542818=22, srv60080846=76, srv1115501922=9, srv1065950540=4, srv1248390958=17, srv957493237=97, srv2077577913=56, srv456289362=70, srv623868796=77, srv97718025=98, srv324126524=65, srv373126717=66, srv901780905=93, srv1168214027=11, srv1049259908=2, srv2087899047=57, srv3062850=64, srv1798172470=49, srv1110024286=7, srv1823215167=51, srv507757146=74, srv1759526055=46, srv882545536=91, srv8574372=88, srv808253903=85, srv1421690070=26, srv2114647104=59, srv1529541612=32, srv1313011017=19, srv850641707=87, srv1511562450=31, srv1633959938=36, srv1341524478=21, srv20640568=55, srv797144342=84, srv1656927160=38, srv464557783=72, srv1015688271=1, srv1883780469=53, srv1466079977=29, srv1279939135=18, srv187703996=52, srv2044580710=54, srv2100383264=58, srv466300415=73, srv1242481169=16, srv688868350=80, srv698295683=81, srv883704493=92, srv996017812=99, srv409842036=68, srv460768953=71, srv1105804300=6, srv1731905308=43, srv1769763198=48, srv229781518=62, srv1206652718=13, srv238374417=63, srv1113585199=8, srv405439271=67, srv1753601754=44} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-30 06:53:25,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-30 06:53:25,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-30 06:53:25,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-30 06:53:25,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-30 06:53:25,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-30 06:53:25,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-30 06:53:25,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-30 06:53:25,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-30 06:53:25,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-30 06:53:25,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-30 06:53:25,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-30 06:53:25,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-30 06:53:25,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-30 06:53:25,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-30 06:53:25,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-30 06:53:25,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-30 06:53:25,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-30 06:53:25,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-30 06:53:25,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-30 06:53:25,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-30 06:53:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-30 06:53:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-30 06:53:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-30 06:53:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-30 06:53:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-30 06:53:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-30 06:53:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-30 06:53:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-30 06:53:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-30 06:53:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-30 06:53:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-30 06:53:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-30 06:53:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-30 06:53:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-30 06:53:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-30 06:53:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-30 06:53:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-30 06:53:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-30 06:53:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-30 06:53:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-30 06:53:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-30 06:53:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-30 06:53:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-30 06:53:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-30 06:53:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-30 06:53:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-30 06:53:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-30 06:53:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-30 06:53:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-30 06:53:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-30 06:53:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-30 06:53:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-30 06:53:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-30 06:53:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-30 06:53:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-30 06:53:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-30 06:53:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-30 06:53:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-30 06:53:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-30 06:53:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-30 06:53:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-30 06:53:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-30 06:53:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-30 06:53:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-30 06:53:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-30 06:53:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-30 06:53:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-30 06:53:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-30 06:53:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-30 06:53:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-30 06:53:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-30 06:53:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-30 06:53:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-30 06:53:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-30 06:53:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-30 06:53:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-30 06:53:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-30 06:53:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-30 06:53:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-30 06:53:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-30 06:53:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-30 06:53:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-30 06:53:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-30 06:53:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-30 06:53:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-30 06:53:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-30 06:53:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-30 06:53:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-30 06:53:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-30 06:53:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-30 06:53:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-30 06:53:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-30 06:53:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-30 06:53:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-30 06:53:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-30 06:53:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-30 06:53:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-30 06:53:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-30 06:53:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-30 06:53:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-30 06:53:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-30 06:53:25,406 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.689421800350782E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 06:53:25,406 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv105985481:83 , srv1420644952:83 , srv1423887042:83 , srv1807714441:83 , srv2087899047:83 , srv2114647104:83 , srv2141993649:83 , srv623868796:83 , srv1015688271:84 , srv1049259908:84 , srv1065950540:84 , srv1098554589:84 , srv1105804300:84 , srv1110024286:84 , srv1113585199:84 , srv1115501922:84 , srv1127904086:84 , srv1168214027:84 , srv1186392218:84 , srv1206652718:84 , srv121676787:84 , srv1224205280:84 , srv1242481169:84 , srv1248390958:84 , srv1279939135:84 , srv1337587997:84 , srv1341524478:84 , srv1353542818:84 , srv1359860626:84 , srv1421690070:84 , srv1439194559:84 , srv1466079977:84 , srv1495573040:84 , srv1511562450:84 , srv1529541612:84 , srv1548886412:84 , srv158276581:84 , srv1602028125:84 , srv1633959938:84 , srv1635652732:84 , srv1656927160:84 , srv166048081:84 , srv1680610345:84 , srv1691811241:84 , srv1731905308:84 , srv1753601754:84 , srv1754329277:84 , srv1759526055:84 , srv1763010278:84 , srv1769763198:84 , srv1798172470:84 , srv1823215167:84 , srv187703996:84 , srv1883780469:84 , srv2044580710:84 , srv20640568:84 , srv2077577913:84 , srv2100383264:84 , srv2123502182:84 , srv229781518:84 , srv3062850:84 , srv324126524:84 , srv373126717:84 , srv405439271:84 , srv409842036:84 , srv432187852:84 , srv456289362:84 , srv460768953:84 , srv466300415:84 , srv507757146:84 , srv571617097:84 , srv60080846:84 , srv624290033:84 , srv669672934:84 , srv688868350:84 , srv698295683:84 , srv704067444:84 , srv797144342:84 , srv808253903:84 , srv84369819:84 , srv850641707:84 , srv8574372:84 , srv869080211:84 , srv87707797:84 , srv882545536:84 , srv901780905:84 , srv908847100:84 , srv918378873:84 , srv928129765:84 , srv957493237:84 , srv97718025:84 , srv996017812:84 , srv1013315937:85 , srv1313011017:85 , srv1657780112:85 , srv238374417:85 , srv464557783:85 , srv732047733:85 , srv883704493:85 , srv1392358657:685 } 2023-05-30 06:53:25,432 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=296 (was 296), ProcessCount=170 (was 170), AvailableMemoryMB=4180 (was 4695) 2023-05-30 06:53:25,442 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=296, ProcessCount=170, AvailableMemoryMB=4180 2023-05-30 06:53:25,442 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-30 06:53:25,443 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-05-30 06:53:25,450 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1019026651:84 , srv1045121057:84 , srv1056986556:84 , srv1071514680:84 , srv1174532653:84 , srv1184089983:84 , srv1186328138:84 , srv1291451407:84 , srv1388949368:84 , srv1400966388:84 , srv1570224794:84 , srv1591478962:84 , srv1605767557:84 , srv1622699396:84 , srv1638646672:84 , srv1665585942:84 , srv171736296:84 , srv1809585410:84 , srv1986472361:84 , srv2013448290:84 , srv2061593469:84 , srv2105750302:84 , srv221707897:84 , srv420613805:84 , srv427531975:84 , srv52037963:84 , srv65481:84 , srv909650558:84 , srv914909893:84 , srv419382383:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-05-30 06:53:25,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-30 06:53:25,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1570224794=10, srv1986472361=18, srv1665585942=15, srv914909893=29, srv1809585410=17, srv909650558=28, srv2105750302=21, srv1056986556=2, srv419382383=23, srv2013448290=19, srv52037963=26, srv1045121057=1, srv1291451407=7, srv1184089983=5, srv1591478962=11, srv1622699396=13, srv1400966388=9, srv171736296=16, srv427531975=25, srv1186328138=6, srv65481=27, srv1638646672=14, srv1605767557=12, srv1174532653=4, srv2061593469=20, srv221707897=22, srv1071514680=3, srv1019026651=0, srv420613805=24, srv1388949368=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-30 06:53:25,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-30 06:53:25,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-30 06:53:25,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-30 06:53:25,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-30 06:53:25,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-30 06:53:25,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-30 06:53:25,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-30 06:53:25,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-30 06:53:25,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-30 06:53:25,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-30 06:53:25,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-30 06:53:25,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-30 06:53:25,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-30 06:53:25,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-30 06:53:25,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-30 06:53:25,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-30 06:53:25,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-30 06:53:25,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-30 06:53:25,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-30 06:53:25,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-30 06:53:25,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-30 06:53:25,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-30 06:53:25,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-30 06:53:25,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-30 06:53:25,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-30 06:53:25,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-30 06:53:25,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-30 06:53:25,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-30 06:53:25,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-30 06:53:25,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-30 06:53:25,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-30 06:53:25,484 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1538884155705105, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 06:53:25,485 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912677940636822, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1538884155705105, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-05-30 06:55:25,467 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6681562 different iterations. Found a solution that moves 1848 regions; Going from a computed imbalance of 0.9912677940636822 to a new imbalance of 4.313324190526491E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6844444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 06:55:25,473 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1019026651:90 , srv1045121057:90 , srv1056986556:90 , srv1071514680:90 , srv1174532653:90 , srv1184089983:90 , srv1186328138:90 , srv1291451407:90 , srv1388949368:90 , srv1400966388:90 , srv1570224794:90 , srv1591478962:90 , srv1605767557:90 , srv1622699396:90 , srv1638646672:90 , srv1665585942:90 , srv171736296:90 , srv1809585410:90 , srv1986472361:90 , srv2013448290:90 , srv2061593469:90 , srv2105750302:90 , srv221707897:90 , srv419382383:90 , srv420613805:90 , srv427531975:90 , srv52037963:90 , srv65481:90 , srv909650558:90 , srv914909893:90 } 2023-05-30 06:55:25,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-30 06:55:25,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1570224794=10, srv1986472361=18, srv1665585942=15, srv914909893=29, srv1809585410=17, srv909650558=28, srv2105750302=21, srv1056986556=2, srv419382383=23, srv2013448290=19, srv52037963=26, srv1045121057=1, srv1291451407=7, srv1184089983=5, srv1591478962=11, srv1622699396=13, srv1400966388=9, srv171736296=16, srv427531975=25, srv1186328138=6, srv65481=27, srv1638646672=14, srv1605767557=12, srv1174532653=4, srv2061593469=20, srv221707897=22, srv1071514680=3, srv1019026651=0, srv420613805=24, srv1388949368=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-30 06:55:25,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-30 06:55:25,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-30 06:55:25,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-30 06:55:25,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-30 06:55:25,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-30 06:55:25,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-30 06:55:25,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-30 06:55:25,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-30 06:55:25,490 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-30 06:55:25,490 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1019026651:90 , srv1045121057:90 , srv1056986556:90 , srv1071514680:90 , srv1174532653:90 , srv1184089983:90 , srv1186328138:90 , srv1291451407:90 , srv1388949368:90 , srv1400966388:90 , srv1570224794:90 , srv1591478962:90 , srv1605767557:90 , srv1622699396:90 , srv1638646672:90 , srv1665585942:90 , srv171736296:90 , srv1809585410:90 , srv1986472361:90 , srv2013448290:90 , srv2061593469:90 , srv2105750302:90 , srv221707897:90 , srv419382383:90 , srv420613805:90 , srv427531975:90 , srv52037963:90 , srv65481:90 , srv909650558:90 , srv914909893:90 } 2023-05-30 06:55:25,502 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=289 (was 296), ProcessCount=168 (was 170), AvailableMemoryMB=3882 (was 4180)