2024-04-10 06:54:49,256 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2024-04-10 06:54:49,532 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-04-10 06:54:49,634 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-04-10 06:54:49,692 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-10 06:54:49,701 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-04-10 06:54:49,715 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=246, ProcessCount=178, AvailableMemoryMB=6851 2024-04-10 06:54:49,717 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-10 06:54:49,717 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2024-04-10 06:54:49,844 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1007887682:84 , srv1022061351:84 , srv1024677896:84 , srv102536389:84 , srv1088998838:84 , srv1112859314:84 , srv1113890162:84 , srv113975137:84 , srv1181435623:84 , srv1193114450:84 , srv1193146240:84 , srv1201157693:84 , srv1207669495:84 , srv1219468020:84 , srv1230799813:84 , srv1249022934:84 , srv1251618170:84 , srv1262572482:84 , srv1326317736:84 , srv1370964898:84 , srv1398591133:84 , srv1404061116:84 , srv1404397793:84 , srv1415581090:84 , srv1417851552:84 , srv1431430950:84 , srv143148086:84 , srv1435445258:84 , srv1444444786:84 , srv1463653797:84 , srv1491494022:84 , srv1493189179:84 , srv1504296943:84 , srv1505015710:84 , srv151474158:84 , srv152729865:84 , srv1540921376:84 , srv1546246123:84 , srv157104542:84 , srv1593969649:84 , srv1604254653:84 , srv1643278294:84 , srv1699484348:84 , srv1715407595:84 , srv1733328321:84 , srv1758613083:84 , srv1774495665:84 , srv180882262:84 , srv1859649092:84 , srv1864536237:84 , srv1865902460:84 , srv1865918188:84 , srv1885000710:84 , srv1908312690:84 , srv1914343375:84 , srv1926258711:84 , srv1941528751:84 , srv1946076529:84 , srv195305912:84 , srv1958271011:84 , srv2016303452:84 , srv2030475025:84 , srv2093205401:84 , srv2102757883:84 , srv2105867394:84 , srv2127601494:84 , srv220392390:84 , srv222214080:84 , srv223049713:84 , srv246666890:84 , srv247026563:84 , srv255020732:84 , srv291138142:84 , srv342757115:84 , srv427584331:84 , srv442782087:84 , srv488147696:84 , srv510356884:84 , srv519285638:84 , srv585456866:84 , srv590378145:84 , srv602859886:84 , srv646391530:84 , srv738177895:84 , srv744534920:84 , srv77450130:84 , srv787428544:84 , srv809020599:84 , srv848699832:84 , srv849538212:84 , srv861507275:84 , srv86782802:84 , srv868366992:84 , srv891122971:84 , srv897037296:84 , srv905395831:84 , srv919139002:84 , srv942257062:84 , srv988717068:84 , srv2019491039:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2024-04-10 06:54:49,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-10 06:54:49,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1593969649=39, srv585456866=80, srv1865918188=51, srv868366992=93, srv220392390=67, srv1715407595=43, srv2127601494=66, srv442782087=76, srv787428544=87, srv891122971=94, srv1643278294=41, srv291138142=73, srv1758613083=45, srv1958271011=59, srv113975137=7, srv1404397793=22, srv1946076529=57, srv1435445258=27, srv1908312690=53, srv143148086=26, srv2030475025=62, srv1370964898=19, srv1417851552=24, srv86782802=92, srv1201157693=11, srv1326317736=18, srv1505015710=33, srv848699832=89, srv246666890=70, srv1926258711=55, srv988717068=99, srv1859649092=48, srv919139002=97, srv1024677896=2, srv1864536237=49, srv1398591133=20, srv1865902460=50, srv222214080=68, srv488147696=77, srv77450130=86, srv510356884=78, srv1699484348=42, srv1540921376=36, srv646391530=83, srv1431430950=25, srv738177895=84, srv1007887682=0, srv2019491039=61, srv255020732=72, srv861507275=91, srv1249022934=15, srv2016303452=60, srv1504296943=32, srv2102757883=64, srv1444444786=28, srv152729865=35, srv427584331=75, srv897037296=95, srv102536389=3, srv157104542=38, srv1262572482=17, srv1415581090=23, srv1491494022=30, srv1493189179=31, srv1181435623=8, srv1251618170=16, srv2093205401=63, srv590378145=81, srv1230799813=14, srv151474158=34, srv1113890162=6, srv1088998838=4, srv1219468020=13, srv342757115=74, srv1463653797=29, srv602859886=82, srv744534920=85, srv849538212=90, srv180882262=47, srv247026563=71, srv1193146240=10, srv1941528751=56, srv519285638=79, srv1733328321=44, srv1112859314=5, srv1774495665=46, srv1193114450=9, srv1207669495=12, srv809020599=88, srv1404061116=21, srv1914343375=54, srv2105867394=65, srv1022061351=1, srv1546246123=37, srv905395831=96, srv1604254653=40, srv195305912=58, srv223049713=69, srv1885000710=52, srv942257062=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-10 06:54:49,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-10 06:54:49,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-10 06:54:49,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-10 06:54:49,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-10 06:54:49,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-10 06:54:49,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-10 06:54:49,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-10 06:54:49,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-10 06:54:49,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-10 06:54:49,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-10 06:54:49,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-10 06:54:49,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-10 06:54:49,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-10 06:54:49,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-10 06:54:49,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-10 06:54:49,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-10 06:54:49,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-10 06:54:49,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-10 06:54:49,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-10 06:54:49,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-10 06:54:49,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-10 06:54:49,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-10 06:54:49,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-10 06:54:49,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-10 06:54:49,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-10 06:54:49,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-10 06:54:49,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-10 06:54:49,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-10 06:54:49,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-10 06:54:49,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-10 06:54:49,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-10 06:54:49,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-10 06:54:49,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-10 06:54:49,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-10 06:54:49,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-10 06:54:49,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-10 06:54:49,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-10 06:54:49,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-10 06:54:49,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-10 06:54:49,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-10 06:54:49,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-10 06:54:49,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-10 06:54:49,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-10 06:54:49,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-10 06:54:49,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-10 06:54:49,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-10 06:54:49,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-10 06:54:49,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-10 06:54:49,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-10 06:54:49,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-10 06:54:49,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-10 06:54:49,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-10 06:54:49,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-10 06:54:49,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-10 06:54:49,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-10 06:54:49,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-10 06:54:49,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-10 06:54:49,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-10 06:54:49,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-10 06:54:49,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-10 06:54:49,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-10 06:54:49,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-10 06:54:49,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-10 06:54:49,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-10 06:54:49,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-10 06:54:49,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-10 06:54:49,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-10 06:54:49,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-10 06:54:49,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-10 06:54:49,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-10 06:54:49,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-10 06:54:49,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-10 06:54:49,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-10 06:54:49,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-10 06:54:49,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-10 06:54:49,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-10 06:54:49,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-10 06:54:49,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-10 06:54:49,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-10 06:54:49,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-10 06:54:49,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-10 06:54:49,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-10 06:54:49,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-10 06:54:49,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-10 06:54:49,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-10 06:54:49,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-10 06:54:49,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-10 06:54:49,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-10 06:54:49,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-10 06:54:49,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-10 06:54:49,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-10 06:54:49,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-10 06:54:49,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-10 06:54:49,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-10 06:54:49,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-10 06:54:49,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-10 06:54:49,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-10 06:54:49,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-10 06:54:49,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-10 06:54:49,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-10 06:54:49,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-10 06:54:50,074 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-10 06:54:50,074 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-10 06:54:50,075 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-10 06:54:50,332 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 382 ms to try 5000 different iterations. Found a solution that moves 4691 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18077052661770757. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03881715365508564, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.048397438041477725, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5212222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03881715365508564, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17558333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24725, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-10 06:54:50,352 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1370964898:81 , srv1493189179:81 , srv1181435623:82 , srv1941528751:82 , srv2127601494:82 , srv891122971:82 , srv897037296:82 , srv1207669495:83 , srv1262572482:83 , srv180882262:83 , srv1885000710:83 , srv1193114450:84 , srv1201157693:84 , srv1417851552:84 , srv151474158:84 , srv1604254653:84 , srv1859649092:84 , srv2102757883:84 , srv861507275:84 , srv86782802:84 , srv988717068:84 , srv1024677896:85 , srv113975137:85 , srv1404397793:85 , srv1733328321:85 , srv1758613083:85 , srv1958271011:85 , srv427584331:85 , srv442782087:85 , srv488147696:85 , srv602859886:85 , srv744534920:85 , srv77450130:85 , srv102536389:86 , srv1251618170:86 , srv1404061116:86 , srv1540921376:86 , srv157104542:86 , srv1715407595:86 , srv1865902460:86 , srv246666890:86 , srv255020732:86 , srv646391530:86 , srv849538212:86 , srv1007887682:87 , srv1088998838:87 , srv1193146240:87 , srv1230799813:87 , srv1326317736:87 , srv1398591133:87 , srv1415581090:87 , srv143148086:87 , srv1504296943:87 , srv152729865:87 , srv1774495665:87 , srv1914343375:87 , srv1926258711:87 , srv1946076529:87 , srv2093205401:87 , srv222214080:87 , srv342757115:87 , srv510356884:87 , srv519285638:87 , srv738177895:87 , srv848699832:87 , srv905395831:87 , srv919139002:87 , srv1113890162:88 , srv1249022934:88 , srv1435445258:88 , srv1444444786:88 , srv1463653797:88 , srv1505015710:88 , srv1864536237:88 , srv2016303452:88 , srv2105867394:88 , srv247026563:88 , srv291138142:88 , srv787428544:88 , srv809020599:88 , srv868366992:88 , srv1219468020:89 , srv1431430950:89 , srv1699484348:89 , srv1865918188:89 , srv220392390:89 , srv223049713:89 , srv585456866:89 , srv1022061351:90 , srv1593969649:90 , srv1643278294:90 , srv195305912:90 , srv2030475025:90 , srv942257062:90 , srv1491494022:91 , srv1908312690:91 , srv1112859314:92 , srv1546246123:92 , srv590378145:93 , srv2019491039:435 } 2024-04-10 06:54:50,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-10 06:54:50,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1593969649=39, srv585456866=80, srv1865918188=51, srv868366992=93, srv220392390=67, srv1715407595=43, srv2127601494=66, srv442782087=76, srv787428544=87, srv891122971=94, srv1643278294=41, srv291138142=73, srv1758613083=45, srv1958271011=59, srv113975137=7, srv1404397793=22, srv1946076529=57, srv1435445258=27, srv1908312690=53, srv143148086=26, srv2030475025=62, srv1370964898=19, srv1417851552=24, srv86782802=92, srv1201157693=11, srv1326317736=18, srv1505015710=33, srv848699832=89, srv246666890=70, srv1926258711=55, srv988717068=99, srv1859649092=48, srv919139002=97, srv1024677896=2, srv1864536237=49, srv1398591133=20, srv1865902460=50, srv222214080=68, srv488147696=77, srv77450130=86, srv510356884=78, srv1699484348=42, srv1540921376=36, srv646391530=83, srv1431430950=25, srv738177895=84, srv1007887682=0, srv2019491039=61, srv255020732=72, srv861507275=91, srv1249022934=15, srv2016303452=60, srv1504296943=32, srv2102757883=64, srv1444444786=28, srv152729865=35, srv427584331=75, srv897037296=95, srv102536389=3, srv157104542=38, srv1262572482=17, srv1415581090=23, srv1491494022=30, srv1493189179=31, srv1181435623=8, srv1251618170=16, srv2093205401=63, srv590378145=81, srv1230799813=14, srv151474158=34, srv1113890162=6, srv1088998838=4, srv1219468020=13, srv342757115=74, srv1463653797=29, srv602859886=82, srv744534920=85, srv849538212=90, srv180882262=47, srv247026563=71, srv1193146240=10, srv1941528751=56, srv519285638=79, srv1733328321=44, srv1112859314=5, srv1774495665=46, srv1193114450=9, srv1207669495=12, srv809020599=88, srv1404061116=21, srv1914343375=54, srv2105867394=65, srv1022061351=1, srv1546246123=37, srv905395831=96, srv1604254653=40, srv195305912=58, srv223049713=69, srv1885000710=52, srv942257062=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-10 06:54:50,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-10 06:54:50,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-10 06:54:50,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-10 06:54:50,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-10 06:54:50,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-10 06:54:50,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-10 06:54:50,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-10 06:54:50,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-10 06:54:50,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-10 06:54:50,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-10 06:54:50,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-10 06:54:50,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-10 06:54:50,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-10 06:54:50,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-10 06:54:50,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-10 06:54:50,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-10 06:54:50,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-10 06:54:50,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-10 06:54:50,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-10 06:54:50,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-10 06:54:50,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-10 06:54:50,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-10 06:54:50,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-10 06:54:50,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-10 06:54:50,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-10 06:54:50,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-10 06:54:50,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-10 06:54:50,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-10 06:54:50,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-10 06:54:50,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-10 06:54:50,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-10 06:54:50,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-10 06:54:50,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-10 06:54:50,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-10 06:54:50,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-10 06:54:50,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-10 06:54:50,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-10 06:54:50,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-10 06:54:50,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-10 06:54:50,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-10 06:54:50,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-10 06:54:50,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-10 06:54:50,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-10 06:54:50,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-10 06:54:50,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-10 06:54:50,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-10 06:54:50,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-10 06:54:50,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-10 06:54:50,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-10 06:54:50,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-10 06:54:50,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-10 06:54:50,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-10 06:54:50,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-10 06:54:50,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-10 06:54:50,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-10 06:54:50,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-10 06:54:50,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-10 06:54:50,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-10 06:54:50,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-10 06:54:50,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-10 06:54:50,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-10 06:54:50,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-10 06:54:50,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-10 06:54:50,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-10 06:54:50,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-10 06:54:50,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-10 06:54:50,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-10 06:54:50,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-10 06:54:50,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-10 06:54:50,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-10 06:54:50,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-10 06:54:50,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-10 06:54:50,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-10 06:54:50,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-10 06:54:50,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-10 06:54:50,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-10 06:54:50,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-10 06:54:50,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-10 06:54:50,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-10 06:54:50,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-10 06:54:50,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-10 06:54:50,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-10 06:54:50,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-10 06:54:50,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-10 06:54:50,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-10 06:54:50,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-10 06:54:50,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-10 06:54:50,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-10 06:54:50,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-10 06:54:50,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-10 06:54:50,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-10 06:54:50,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-10 06:54:50,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-10 06:54:50,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-10 06:54:50,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-10 06:54:50,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-10 06:54:50,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-10 06:54:50,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-10 06:54:50,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-10 06:54:50,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-10 06:54:50,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-10 06:54:50,493 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03881715365508564, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.048397438041477725, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03881715365508564, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17558333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24725, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-10 06:54:50,493 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-10 06:54:50,493 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18073767953365275, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03881715365508564, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.048397438041477725, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03881715365508564, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17558333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24725, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-10 06:54:50,661 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 247 ms to try 5000 different iterations. Found a solution that moves 2680 regions; Going from a computed imbalance of 0.18073767953365275 to a new imbalance of 0.016749983619073712. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02227882801737244); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03140195937755667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29777777777777775, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02227882801737244); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-10 06:54:50,671 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1546246123:79 , srv590378145:79 , srv195305912:80 , srv1431430950:81 , srv1491494022:81 , srv1643278294:81 , srv1865918188:81 , srv1113890162:82 , srv1593969649:82 , srv2030475025:82 , srv1112859314:83 , srv1398591133:83 , srv1435445258:83 , srv1864536237:83 , srv1908312690:83 , srv1946076529:83 , srv220392390:83 , srv519285638:83 , srv585456866:83 , srv809020599:83 , srv942257062:83 , srv1022061351:84 , srv1088998838:84 , srv1249022934:84 , srv1415581090:84 , srv1444444786:84 , srv1540921376:84 , srv1699484348:84 , srv1926258711:84 , srv222214080:84 , srv510356884:84 , srv646391530:84 , srv787428544:84 , srv849538212:84 , srv868366992:84 , srv919139002:84 , srv102536389:85 , srv1219468020:85 , srv1326317736:85 , srv1504296943:85 , srv1505015710:85 , srv152729865:85 , srv1733328321:85 , srv1774495665:85 , srv1865902460:85 , srv1958271011:85 , srv2105867394:85 , srv223049713:85 , srv246666890:85 , srv247026563:85 , srv291138142:85 , srv427584331:85 , srv988717068:85 , srv1007887682:86 , srv113975137:86 , srv1193146240:86 , srv1230799813:86 , srv1463653797:86 , srv157104542:86 , srv1715407595:86 , srv1914343375:86 , srv2016303452:86 , srv2093205401:86 , srv2102757883:86 , srv342757115:86 , srv738177895:86 , srv848699832:86 , srv905395831:86 , srv1201157693:87 , srv1404061116:87 , srv143148086:87 , srv1604254653:87 , srv1859649092:87 , srv255020732:87 , srv442782087:87 , srv602859886:87 , srv744534920:87 , srv861507275:87 , srv1024677896:88 , srv1193114450:88 , srv1251618170:88 , srv1493189179:88 , srv1941528751:88 , srv1181435623:89 , srv1207669495:89 , srv1404397793:89 , srv1417851552:89 , srv1758613083:89 , srv180882262:89 , srv77450130:89 , srv897037296:89 , srv488147696:90 , srv86782802:90 , srv891122971:90 , srv1262572482:91 , srv1370964898:91 , srv151474158:91 , srv1885000710:91 , srv2127601494:91 , srv2019491039:537 } 2024-04-10 06:54:50,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-10 06:54:50,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1593969649=39, srv585456866=80, srv1865918188=51, srv868366992=93, srv220392390=67, srv1715407595=43, srv2127601494=66, srv442782087=76, srv787428544=87, srv891122971=94, srv1643278294=41, srv291138142=73, srv1758613083=45, srv1958271011=59, srv113975137=7, srv1404397793=22, srv1946076529=57, srv1435445258=27, srv1908312690=53, srv143148086=26, srv2030475025=62, srv1370964898=19, srv1417851552=24, srv86782802=92, srv1201157693=11, srv1326317736=18, srv1505015710=33, srv848699832=89, srv246666890=70, srv1926258711=55, srv988717068=99, srv1859649092=48, srv919139002=97, srv1024677896=2, srv1864536237=49, srv1398591133=20, srv1865902460=50, srv222214080=68, srv488147696=77, srv77450130=86, srv510356884=78, srv1699484348=42, srv1540921376=36, srv646391530=83, srv1431430950=25, srv738177895=84, srv1007887682=0, srv2019491039=61, srv255020732=72, srv861507275=91, srv1249022934=15, srv2016303452=60, srv1504296943=32, srv2102757883=64, srv1444444786=28, srv152729865=35, srv427584331=75, srv897037296=95, srv102536389=3, srv157104542=38, srv1262572482=17, srv1415581090=23, srv1491494022=30, srv1493189179=31, srv1181435623=8, srv1251618170=16, srv2093205401=63, srv590378145=81, srv1230799813=14, srv151474158=34, srv1113890162=6, srv1088998838=4, srv1219468020=13, srv342757115=74, srv1463653797=29, srv602859886=82, srv744534920=85, srv849538212=90, srv180882262=47, srv247026563=71, srv1193146240=10, srv1941528751=56, srv519285638=79, srv1733328321=44, srv1112859314=5, srv1774495665=46, srv1193114450=9, srv1207669495=12, srv809020599=88, srv1404061116=21, srv1914343375=54, srv2105867394=65, srv1022061351=1, srv1546246123=37, srv905395831=96, srv1604254653=40, srv195305912=58, srv223049713=69, srv1885000710=52, srv942257062=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-10 06:54:50,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-10 06:54:50,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-10 06:54:50,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-10 06:54:50,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-10 06:54:50,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-10 06:54:50,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-10 06:54:50,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-10 06:54:50,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-10 06:54:50,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-10 06:54:50,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-10 06:54:50,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-10 06:54:50,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-10 06:54:50,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-10 06:54:50,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-10 06:54:50,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-10 06:54:50,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-10 06:54:50,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-10 06:54:50,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-10 06:54:50,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-10 06:54:50,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-10 06:54:50,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-10 06:54:50,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-10 06:54:50,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-10 06:54:50,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-10 06:54:50,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-10 06:54:50,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-10 06:54:50,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-10 06:54:50,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-10 06:54:50,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-10 06:54:50,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-10 06:54:50,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-10 06:54:50,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-10 06:54:50,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-10 06:54:50,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-10 06:54:50,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-10 06:54:50,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-10 06:54:50,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-10 06:54:50,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-10 06:54:50,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-10 06:54:50,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-10 06:54:50,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-10 06:54:50,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-10 06:54:50,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-10 06:54:50,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-10 06:54:50,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-10 06:54:50,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-10 06:54:50,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-10 06:54:50,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-10 06:54:50,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-10 06:54:50,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-10 06:54:50,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-10 06:54:50,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-10 06:54:50,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-10 06:54:50,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-10 06:54:50,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-10 06:54:50,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-10 06:54:50,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-10 06:54:50,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-10 06:54:50,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-10 06:54:50,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-10 06:54:50,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-10 06:54:50,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-10 06:54:50,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-10 06:54:50,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-10 06:54:50,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-10 06:54:50,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-10 06:54:50,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-10 06:54:50,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-10 06:54:50,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-10 06:54:50,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-10 06:54:50,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-10 06:54:50,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-10 06:54:50,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-10 06:54:50,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-10 06:54:50,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-10 06:54:50,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-10 06:54:50,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-10 06:54:50,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-10 06:54:50,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-10 06:54:50,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-10 06:54:50,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-10 06:54:50,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-10 06:54:50,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-10 06:54:50,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-10 06:54:50,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-10 06:54:50,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-10 06:54:50,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-10 06:54:50,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-10 06:54:50,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-10 06:54:50,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-10 06:54:50,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-10 06:54:50,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-10 06:54:50,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-10 06:54:50,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-10 06:54:50,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-10 06:54:50,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-10 06:54:50,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-10 06:54:50,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-10 06:54:50,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-10 06:54:50,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-10 06:54:50,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-10 06:54:50,814 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02227882801737244); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03140195937755667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02227882801737244); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-10 06:54:50,814 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-10 06:54:50,814 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016731217857985058, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02227882801737244); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03140195937755667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02227882801737244); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-10 06:54:50,978 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 250 ms to try 5000 different iterations. Found a solution that moves 998 regions; Going from a computed imbalance of 0.016731217857985058 to a new imbalance of 6.875991046982183E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008461970117626564); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008812805701489793); MoveCostFunction : (multiplier=7.0, imbalance=0.11088888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008461970117626564); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-10 06:54:50,980 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1112859314:81 , srv1022061351:83 , srv1758613083:83 , srv1885000710:83 , srv1908312690:83 , srv2016303452:83 , srv223049713:83 , srv77450130:83 , srv86782802:83 , srv1024677896:84 , srv1193114450:84 , srv1219468020:84 , srv1230799813:84 , srv1249022934:84 , srv1262572482:84 , srv1404397793:84 , srv143148086:84 , srv1463653797:84 , srv1505015710:84 , srv151474158:84 , srv180882262:84 , srv1914343375:84 , srv2093205401:84 , srv2105867394:84 , srv247026563:84 , srv255020732:84 , srv291138142:84 , srv488147696:84 , srv602859886:84 , srv848699832:84 , srv868366992:84 , srv891122971:84 , srv1007887682:85 , srv1088998838:85 , srv1193146240:85 , srv1207669495:85 , srv1251618170:85 , srv1370964898:85 , srv1404061116:85 , srv1415581090:85 , srv1444444786:85 , srv1491494022:85 , srv152729865:85 , srv157104542:85 , srv1593969649:85 , srv1699484348:85 , srv1715407595:85 , srv1774495665:85 , srv1859649092:85 , srv1926258711:85 , srv2030475025:85 , srv2127601494:85 , srv220392390:85 , srv585456866:85 , srv590378145:85 , srv738177895:85 , srv744534920:85 , srv787428544:85 , srv905395831:85 , srv942257062:85 , srv113975137:86 , srv1181435623:86 , srv1201157693:86 , srv1326317736:86 , srv1398591133:86 , srv1417851552:86 , srv1435445258:86 , srv1504296943:86 , srv1540921376:86 , srv1546246123:86 , srv1864536237:86 , srv195305912:86 , srv222214080:86 , srv342757115:86 , srv442782087:86 , srv510356884:86 , srv519285638:86 , srv809020599:86 , srv861507275:86 , srv102536389:87 , srv1113890162:87 , srv1431430950:87 , srv1604254653:87 , srv1733328321:87 , srv1865902460:87 , srv1865918188:87 , srv1946076529:87 , srv246666890:87 , srv427584331:87 , srv646391530:87 , srv849538212:87 , srv897037296:87 , srv919139002:87 , srv1643278294:88 , srv1941528751:88 , srv1958271011:88 , srv2102757883:88 , srv1493189179:89 , srv988717068:89 , srv2019491039:561 } 2024-04-10 06:54:50,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-10 06:54:50,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1593969649=39, srv585456866=80, srv1865918188=51, srv868366992=93, srv220392390=67, srv1715407595=43, srv2127601494=66, srv442782087=76, srv787428544=87, srv891122971=94, srv1643278294=41, srv291138142=73, srv1758613083=45, srv1958271011=59, srv113975137=7, srv1404397793=22, srv1946076529=57, srv1435445258=27, srv1908312690=53, srv143148086=26, srv2030475025=62, srv1370964898=19, srv1417851552=24, srv86782802=92, srv1201157693=11, srv1326317736=18, srv1505015710=33, srv848699832=89, srv246666890=70, srv1926258711=55, srv988717068=99, srv1859649092=48, srv919139002=97, srv1024677896=2, srv1864536237=49, srv1398591133=20, srv1865902460=50, srv222214080=68, srv488147696=77, srv77450130=86, srv510356884=78, srv1699484348=42, srv1540921376=36, srv646391530=83, srv1431430950=25, srv738177895=84, srv1007887682=0, srv2019491039=61, srv255020732=72, srv861507275=91, srv1249022934=15, srv2016303452=60, srv1504296943=32, srv2102757883=64, srv1444444786=28, srv152729865=35, srv427584331=75, srv897037296=95, srv102536389=3, srv157104542=38, srv1262572482=17, srv1415581090=23, srv1491494022=30, srv1493189179=31, srv1181435623=8, srv1251618170=16, srv2093205401=63, srv590378145=81, srv1230799813=14, srv151474158=34, srv1113890162=6, srv1088998838=4, srv1219468020=13, srv342757115=74, srv1463653797=29, srv602859886=82, srv744534920=85, srv849538212=90, srv180882262=47, srv247026563=71, srv1193146240=10, srv1941528751=56, srv519285638=79, srv1733328321=44, srv1112859314=5, srv1774495665=46, srv1193114450=9, srv1207669495=12, srv809020599=88, srv1404061116=21, srv1914343375=54, srv2105867394=65, srv1022061351=1, srv1546246123=37, srv905395831=96, srv1604254653=40, srv195305912=58, srv223049713=69, srv1885000710=52, srv942257062=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-10 06:54:50,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-10 06:54:50,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-10 06:54:50,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-10 06:54:50,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-10 06:54:50,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-10 06:54:50,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-10 06:54:50,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-10 06:54:50,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-10 06:54:50,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-10 06:54:50,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-10 06:54:50,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-10 06:54:50,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-10 06:54:50,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-10 06:54:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-10 06:54:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-10 06:54:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-10 06:54:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-10 06:54:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-10 06:54:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-10 06:54:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-10 06:54:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-10 06:54:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-10 06:54:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-10 06:54:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-10 06:54:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-10 06:54:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-10 06:54:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-10 06:54:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-10 06:54:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-10 06:54:50,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-10 06:54:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-10 06:54:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-10 06:54:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-10 06:54:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-10 06:54:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-10 06:54:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-10 06:54:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-10 06:54:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-10 06:54:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-10 06:54:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-10 06:54:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-10 06:54:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-10 06:54:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-10 06:54:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-10 06:54:50,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-10 06:54:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-10 06:54:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-10 06:54:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-10 06:54:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-10 06:54:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-10 06:54:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-10 06:54:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-10 06:54:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-10 06:54:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-10 06:54:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-10 06:54:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-10 06:54:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-10 06:54:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-10 06:54:50,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-10 06:54:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-10 06:54:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-10 06:54:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-10 06:54:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-10 06:54:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-10 06:54:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-10 06:54:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-10 06:54:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-10 06:54:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-10 06:54:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-10 06:54:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-10 06:54:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-10 06:54:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-10 06:54:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-10 06:54:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-10 06:54:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-10 06:54:50,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-10 06:54:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-10 06:54:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-10 06:54:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-10 06:54:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-10 06:54:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-10 06:54:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-10 06:54:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-10 06:54:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-10 06:54:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-10 06:54:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-10 06:54:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-10 06:54:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-10 06:54:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-10 06:54:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-10 06:54:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-10 06:54:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-10 06:54:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-10 06:54:50,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-10 06:54:50,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-10 06:54:50,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-10 06:54:50,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-10 06:54:50,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-10 06:54:50,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-10 06:54:50,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-10 06:54:50,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-10 06:54:51,124 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008461970117626564); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008812805701489793); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008461970117626564); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-10 06:54:51,124 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-10 06:54:51,124 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.806109593375926E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008461970117626564); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008812805701489793); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008461970117626564); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-10 06:54:51,398 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 386 ms to try 5000 different iterations. Found a solution that moves 286 regions; Going from a computed imbalance of 6.806109593375926E-4 to a new imbalance of 5.8835936189871246E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.03177777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-10 06:54:51,399 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1251618170:83 , srv1417851552:83 , srv1643278294:83 , srv2102757883:83 , srv342757115:83 , srv488147696:83 , srv1007887682:84 , srv102536389:84 , srv1193146240:84 , srv1219468020:84 , srv1230799813:84 , srv1262572482:84 , srv1326317736:84 , srv1404397793:84 , srv143148086:84 , srv1463653797:84 , srv1493189179:84 , srv1504296943:84 , srv151474158:84 , srv1593969649:84 , srv1699484348:84 , srv1865902460:84 , srv1941528751:84 , srv1958271011:84 , srv2127601494:84 , srv220392390:84 , srv246666890:84 , srv247026563:84 , srv442782087:84 , srv738177895:84 , srv897037296:84 , srv919139002:84 , srv942257062:84 , srv988717068:84 , srv1022061351:85 , srv1024677896:85 , srv1088998838:85 , srv1113890162:85 , srv113975137:85 , srv1193114450:85 , srv1201157693:85 , srv1207669495:85 , srv1370964898:85 , srv1404061116:85 , srv1431430950:85 , srv1435445258:85 , srv1444444786:85 , srv1491494022:85 , srv1505015710:85 , srv152729865:85 , srv1546246123:85 , srv157104542:85 , srv1604254653:85 , srv1715407595:85 , srv1733328321:85 , srv1758613083:85 , srv1774495665:85 , srv1864536237:85 , srv1865918188:85 , srv1885000710:85 , srv1908312690:85 , srv1914343375:85 , srv1946076529:85 , srv195305912:85 , srv2016303452:85 , srv2030475025:85 , srv2105867394:85 , srv222214080:85 , srv223049713:85 , srv255020732:85 , srv291138142:85 , srv427584331:85 , srv510356884:85 , srv585456866:85 , srv590378145:85 , srv646391530:85 , srv744534920:85 , srv77450130:85 , srv809020599:85 , srv849538212:85 , srv861507275:85 , srv86782802:85 , srv891122971:85 , srv905395831:85 , srv1112859314:86 , srv1181435623:86 , srv1249022934:86 , srv1398591133:86 , srv1415581090:86 , srv1540921376:86 , srv180882262:86 , srv1859649092:86 , srv1926258711:86 , srv2093205401:86 , srv519285638:86 , srv602859886:86 , srv787428544:86 , srv848699832:86 , srv868366992:86 , srv2019491039:610 } 2024-04-10 06:54:51,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-10 06:54:51,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1593969649=39, srv585456866=80, srv1865918188=51, srv868366992=93, srv220392390=67, srv1715407595=43, srv2127601494=66, srv442782087=76, srv787428544=87, srv891122971=94, srv1643278294=41, srv291138142=73, srv1758613083=45, srv1958271011=59, srv113975137=7, srv1404397793=22, srv1946076529=57, srv1435445258=27, srv1908312690=53, srv143148086=26, srv2030475025=62, srv1370964898=19, srv1417851552=24, srv86782802=92, srv1201157693=11, srv1326317736=18, srv1505015710=33, srv848699832=89, srv246666890=70, srv1926258711=55, srv988717068=99, srv1859649092=48, srv919139002=97, srv1024677896=2, srv1864536237=49, srv1398591133=20, srv1865902460=50, srv222214080=68, srv488147696=77, srv77450130=86, srv510356884=78, srv1699484348=42, srv1540921376=36, srv646391530=83, srv1431430950=25, srv738177895=84, srv1007887682=0, srv2019491039=61, srv255020732=72, srv861507275=91, srv1249022934=15, srv2016303452=60, srv1504296943=32, srv2102757883=64, srv1444444786=28, srv152729865=35, srv427584331=75, srv897037296=95, srv102536389=3, srv157104542=38, srv1262572482=17, srv1415581090=23, srv1491494022=30, srv1493189179=31, srv1181435623=8, srv1251618170=16, srv2093205401=63, srv590378145=81, srv1230799813=14, srv151474158=34, srv1113890162=6, srv1088998838=4, srv1219468020=13, srv342757115=74, srv1463653797=29, srv602859886=82, srv744534920=85, srv849538212=90, srv180882262=47, srv247026563=71, srv1193146240=10, srv1941528751=56, srv519285638=79, srv1733328321=44, srv1112859314=5, srv1774495665=46, srv1193114450=9, srv1207669495=12, srv809020599=88, srv1404061116=21, srv1914343375=54, srv2105867394=65, srv1022061351=1, srv1546246123=37, srv905395831=96, srv1604254653=40, srv195305912=58, srv223049713=69, srv1885000710=52, srv942257062=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-10 06:54:51,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-10 06:54:51,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-10 06:54:51,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-10 06:54:51,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-10 06:54:51,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-10 06:54:51,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-10 06:54:51,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-10 06:54:51,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-10 06:54:51,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-10 06:54:51,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-10 06:54:51,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-10 06:54:51,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-10 06:54:51,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-10 06:54:51,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-10 06:54:51,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-10 06:54:51,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-10 06:54:51,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-10 06:54:51,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-10 06:54:51,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-10 06:54:51,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-10 06:54:51,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-10 06:54:51,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-10 06:54:51,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-10 06:54:51,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-10 06:54:51,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-10 06:54:51,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-10 06:54:51,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-10 06:54:51,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-10 06:54:51,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-10 06:54:51,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-10 06:54:51,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-10 06:54:51,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-10 06:54:51,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-10 06:54:51,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-10 06:54:51,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-10 06:54:51,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-10 06:54:51,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-10 06:54:51,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-10 06:54:51,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-10 06:54:51,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-10 06:54:51,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-10 06:54:51,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-10 06:54:51,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-10 06:54:51,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-10 06:54:51,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-10 06:54:51,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-10 06:54:51,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-10 06:54:51,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-10 06:54:51,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-10 06:54:51,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-10 06:54:51,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-10 06:54:51,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-10 06:54:51,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-10 06:54:51,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-10 06:54:51,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-10 06:54:51,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-10 06:54:51,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-10 06:54:51,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-10 06:54:51,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-10 06:54:51,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-10 06:54:51,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-10 06:54:51,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-10 06:54:51,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-10 06:54:51,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-10 06:54:51,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-10 06:54:51,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-10 06:54:51,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-10 06:54:51,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-10 06:54:51,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-10 06:54:51,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-10 06:54:51,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-10 06:54:51,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-10 06:54:51,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-10 06:54:51,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-10 06:54:51,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-10 06:54:51,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-10 06:54:51,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-10 06:54:51,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-10 06:54:51,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-10 06:54:51,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-10 06:54:51,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-10 06:54:51,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-10 06:54:51,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-10 06:54:51,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-10 06:54:51,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-10 06:54:51,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-10 06:54:51,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-10 06:54:51,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-10 06:54:51,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-10 06:54:51,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-10 06:54:51,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-10 06:54:51,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-10 06:54:51,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-10 06:54:51,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-10 06:54:51,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-10 06:54:51,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-10 06:54:51,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-10 06:54:51,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-10 06:54:51,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-10 06:54:51,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-10 06:54:51,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-10 06:54:51,536 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-10 06:54:51,537 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2024-04-10 06:54:51,537 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.68333213871268E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2024-04-10 06:54:51,686 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 245 ms to try 5000 different iterations. Found a solution that moves 48 regions; Going from a computed imbalance of 5.68333213871268E-5 to a new imbalance of 6.368181923405758E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0013400504203456161); MoveCostFunction : (multiplier=7.0, imbalance=0.005333333333333333); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-10 06:54:51,687 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1181435623:83 , srv1404061116:83 , srv151474158:83 , srv1604254653:83 , srv2093205401:83 , srv787428544:83 , srv848699832:83 , srv905395831:83 , srv1007887682:84 , srv1022061351:84 , srv1024677896:84 , srv102536389:84 , srv1112859314:84 , srv1113890162:84 , srv113975137:84 , srv1193146240:84 , srv1201157693:84 , srv1207669495:84 , srv1219468020:84 , srv1249022934:84 , srv1251618170:84 , srv1262572482:84 , srv1326317736:84 , srv1370964898:84 , srv1398591133:84 , srv1404397793:84 , srv1415581090:84 , srv1417851552:84 , srv1431430950:84 , srv143148086:84 , srv1435445258:84 , srv1444444786:84 , srv1463653797:84 , srv1491494022:84 , srv1493189179:84 , srv1504296943:84 , srv1505015710:84 , srv152729865:84 , srv1540921376:84 , srv1546246123:84 , srv157104542:84 , srv1643278294:84 , srv1699484348:84 , srv1715407595:84 , srv1733328321:84 , srv1758613083:84 , srv1774495665:84 , srv180882262:84 , srv1859649092:84 , srv1864536237:84 , srv1865902460:84 , srv1865918188:84 , srv1885000710:84 , srv1908312690:84 , srv1914343375:84 , srv1926258711:84 , srv1941528751:84 , srv1946076529:84 , srv1958271011:84 , srv2016303452:84 , srv2030475025:84 , srv2105867394:84 , srv2127601494:84 , srv222214080:84 , srv223049713:84 , srv246666890:84 , srv255020732:84 , srv291138142:84 , srv342757115:84 , srv427584331:84 , srv442782087:84 , srv488147696:84 , srv510356884:84 , srv519285638:84 , srv585456866:84 , srv590378145:84 , srv602859886:84 , srv646391530:84 , srv738177895:84 , srv744534920:84 , srv77450130:84 , srv809020599:84 , srv849538212:84 , srv861507275:84 , srv86782802:84 , srv868366992:84 , srv897037296:84 , srv919139002:84 , srv942257062:84 , srv988717068:84 , srv1088998838:85 , srv1193114450:85 , srv1230799813:85 , srv1593969649:85 , srv195305912:85 , srv2102757883:85 , srv220392390:85 , srv247026563:85 , srv891122971:85 , srv2019491039:683 } 2024-04-10 06:54:51,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-10 06:54:51,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1593969649=39, srv585456866=80, srv1865918188=51, srv868366992=93, srv220392390=67, srv1715407595=43, srv2127601494=66, srv442782087=76, srv787428544=87, srv891122971=94, srv1643278294=41, srv291138142=73, srv1758613083=45, srv1958271011=59, srv113975137=7, srv1404397793=22, srv1946076529=57, srv1435445258=27, srv1908312690=53, srv143148086=26, srv2030475025=62, srv1370964898=19, srv1417851552=24, srv86782802=92, srv1201157693=11, srv1326317736=18, srv1505015710=33, srv848699832=89, srv246666890=70, srv1926258711=55, srv988717068=99, srv1859649092=48, srv919139002=97, srv1024677896=2, srv1864536237=49, srv1398591133=20, srv1865902460=50, srv222214080=68, srv488147696=77, srv77450130=86, srv510356884=78, srv1699484348=42, srv1540921376=36, srv646391530=83, srv1431430950=25, srv738177895=84, srv1007887682=0, srv2019491039=61, srv255020732=72, srv861507275=91, srv1249022934=15, srv2016303452=60, srv1504296943=32, srv2102757883=64, srv1444444786=28, srv152729865=35, srv427584331=75, srv897037296=95, srv102536389=3, srv157104542=38, srv1262572482=17, srv1415581090=23, srv1491494022=30, srv1493189179=31, srv1181435623=8, srv1251618170=16, srv2093205401=63, srv590378145=81, srv1230799813=14, srv151474158=34, srv1113890162=6, srv1088998838=4, srv1219468020=13, srv342757115=74, srv1463653797=29, srv602859886=82, srv744534920=85, srv849538212=90, srv180882262=47, srv247026563=71, srv1193146240=10, srv1941528751=56, srv519285638=79, srv1733328321=44, srv1112859314=5, srv1774495665=46, srv1193114450=9, srv1207669495=12, srv809020599=88, srv1404061116=21, srv1914343375=54, srv2105867394=65, srv1022061351=1, srv1546246123=37, srv905395831=96, srv1604254653=40, srv195305912=58, srv223049713=69, srv1885000710=52, srv942257062=98} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-10 06:54:51,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-10 06:54:51,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-10 06:54:51,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-10 06:54:51,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-10 06:54:51,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-10 06:54:51,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-10 06:54:51,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-10 06:54:51,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-10 06:54:51,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-10 06:54:51,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-10 06:54:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-10 06:54:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-10 06:54:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-10 06:54:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-10 06:54:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-10 06:54:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-10 06:54:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-10 06:54:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-10 06:54:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-10 06:54:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-10 06:54:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-10 06:54:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-10 06:54:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-10 06:54:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-10 06:54:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-10 06:54:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-10 06:54:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-10 06:54:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-10 06:54:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-10 06:54:51,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-10 06:54:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2024-04-10 06:54:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2024-04-10 06:54:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2024-04-10 06:54:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2024-04-10 06:54:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2024-04-10 06:54:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2024-04-10 06:54:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2024-04-10 06:54:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2024-04-10 06:54:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2024-04-10 06:54:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2024-04-10 06:54:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2024-04-10 06:54:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2024-04-10 06:54:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2024-04-10 06:54:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2024-04-10 06:54:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2024-04-10 06:54:51,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2024-04-10 06:54:51,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2024-04-10 06:54:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2024-04-10 06:54:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2024-04-10 06:54:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2024-04-10 06:54:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2024-04-10 06:54:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2024-04-10 06:54:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2024-04-10 06:54:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2024-04-10 06:54:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2024-04-10 06:54:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2024-04-10 06:54:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2024-04-10 06:54:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2024-04-10 06:54:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2024-04-10 06:54:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2024-04-10 06:54:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2024-04-10 06:54:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2024-04-10 06:54:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2024-04-10 06:54:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2024-04-10 06:54:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2024-04-10 06:54:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2024-04-10 06:54:51,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2024-04-10 06:54:51,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2024-04-10 06:54:51,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2024-04-10 06:54:51,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2024-04-10 06:54:51,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2024-04-10 06:54:51,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2024-04-10 06:54:51,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2024-04-10 06:54:51,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2024-04-10 06:54:51,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2024-04-10 06:54:51,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2024-04-10 06:54:51,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2024-04-10 06:54:51,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2024-04-10 06:54:51,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2024-04-10 06:54:51,789 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.032078739728369E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0013400504203456161); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-10 06:54:51,789 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1181435623:83 , srv1404061116:83 , srv151474158:83 , srv1604254653:83 , srv2093205401:83 , srv787428544:83 , srv848699832:83 , srv905395831:83 , srv1007887682:84 , srv1022061351:84 , srv1024677896:84 , srv102536389:84 , srv1112859314:84 , srv1113890162:84 , srv113975137:84 , srv1193146240:84 , srv1201157693:84 , srv1207669495:84 , srv1219468020:84 , srv1249022934:84 , srv1251618170:84 , srv1262572482:84 , srv1326317736:84 , srv1370964898:84 , srv1398591133:84 , srv1404397793:84 , srv1415581090:84 , srv1417851552:84 , srv1431430950:84 , srv143148086:84 , srv1435445258:84 , srv1444444786:84 , srv1463653797:84 , srv1491494022:84 , srv1493189179:84 , srv1504296943:84 , srv1505015710:84 , srv152729865:84 , srv1540921376:84 , srv1546246123:84 , srv157104542:84 , srv1643278294:84 , srv1699484348:84 , srv1715407595:84 , srv1733328321:84 , srv1758613083:84 , srv1774495665:84 , srv180882262:84 , srv1859649092:84 , srv1864536237:84 , srv1865902460:84 , srv1865918188:84 , srv1885000710:84 , srv1908312690:84 , srv1914343375:84 , srv1926258711:84 , srv1941528751:84 , srv1946076529:84 , srv1958271011:84 , srv2016303452:84 , srv2030475025:84 , srv2105867394:84 , srv2127601494:84 , srv222214080:84 , srv223049713:84 , srv246666890:84 , srv255020732:84 , srv291138142:84 , srv342757115:84 , srv427584331:84 , srv442782087:84 , srv488147696:84 , srv510356884:84 , srv519285638:84 , srv585456866:84 , srv590378145:84 , srv602859886:84 , srv646391530:84 , srv738177895:84 , srv744534920:84 , srv77450130:84 , srv809020599:84 , srv849538212:84 , srv861507275:84 , srv86782802:84 , srv868366992:84 , srv897037296:84 , srv919139002:84 , srv942257062:84 , srv988717068:84 , srv1088998838:85 , srv1193114450:85 , srv1230799813:85 , srv1593969649:85 , srv195305912:85 , srv2102757883:85 , srv220392390:85 , srv247026563:85 , srv891122971:85 , srv2019491039:683 } 2024-04-10 06:54:51,812 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=246 (was 246), ProcessCount=180 (was 178) - ProcessCount LEAK? -, AvailableMemoryMB=6013 (was 6851) 2024-04-10 06:54:51,820 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=246, ProcessCount=180, AvailableMemoryMB=6013 2024-04-10 06:54:51,821 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2024-04-10 06:54:51,821 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2024-04-10 06:54:51,831 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1100373233:84 , srv1125160257:84 , srv1149371336:84 , srv1153246207:84 , srv1153515720:84 , srv1240267933:84 , srv1552637476:84 , srv1556299904:84 , srv1655510331:84 , srv1674146292:84 , srv1766907952:84 , srv1898635254:84 , srv1935386394:84 , srv227155055:84 , srv258406347:84 , srv272885343:84 , srv481017892:84 , srv51663951:84 , srv624036704:84 , srv63621446:84 , srv637396533:84 , srv712347512:84 , srv733551302:84 , srv74212547:84 , srv794879293:84 , srv835937613:84 , srv893212801:84 , srv939556074:84 , srv979750212:84 , srv1030550433:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2024-04-10 06:54:51,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-10 06:54:51,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1898635254=12, srv1240267933=6, srv63621446=20, srv1766907952=11, srv1030550433=0, srv637396533=21, srv1552637476=7, srv227155055=14, srv624036704=19, srv893212801=27, srv1935386394=13, srv1655510331=9, srv733551302=23, srv712347512=22, srv272885343=16, srv74212547=24, srv794879293=25, srv1100373233=1, srv1674146292=10, srv1149371336=3, srv258406347=15, srv939556074=28, srv1153515720=5, srv1153246207=4, srv835937613=26, srv51663951=18, srv1125160257=2, srv481017892=17, srv1556299904=8, srv979750212=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-10 06:54:51,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-10 06:54:51,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-10 06:54:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-10 06:54:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-10 06:54:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-10 06:54:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-10 06:54:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-10 06:54:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-10 06:54:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-10 06:54:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-10 06:54:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-10 06:54:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-10 06:54:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-10 06:54:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-10 06:54:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-10 06:54:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-10 06:54:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-10 06:54:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-10 06:54:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-10 06:54:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-10 06:54:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-10 06:54:51,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-10 06:54:51,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-10 06:54:51,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-10 06:54:51,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-10 06:54:51,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-10 06:54:51,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-10 06:54:51,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-10 06:54:51,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-10 06:54:51,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-10 06:54:51,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-04-10 06:54:51,855 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1158578042433125, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-10 06:54:51,855 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912558107422345, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1158578042433125, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2024-04-10 06:56:51,845 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9645176 different iterations. Found a solution that moves 1844 regions; Going from a computed imbalance of 0.9912558107422345 to a new imbalance of 4.303987990979898E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.682962962962963, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-10 06:56:51,850 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1030550433:90 , srv1100373233:90 , srv1125160257:90 , srv1149371336:90 , srv1153246207:90 , srv1153515720:90 , srv1240267933:90 , srv1552637476:90 , srv1556299904:90 , srv1655510331:90 , srv1674146292:90 , srv1766907952:90 , srv1898635254:90 , srv1935386394:90 , srv227155055:90 , srv258406347:90 , srv272885343:90 , srv481017892:90 , srv51663951:90 , srv624036704:90 , srv63621446:90 , srv637396533:90 , srv712347512:90 , srv733551302:90 , srv74212547:90 , srv794879293:90 , srv835937613:90 , srv893212801:90 , srv939556074:90 , srv979750212:90 } 2024-04-10 06:56:51,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2024-04-10 06:56:51,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1898635254=12, srv1240267933=6, srv63621446=20, srv1766907952=11, srv1030550433=0, srv637396533=21, srv1552637476=7, srv227155055=14, srv624036704=19, srv893212801=27, srv1935386394=13, srv1655510331=9, srv733551302=23, srv712347512=22, srv272885343=16, srv74212547=24, srv794879293=25, srv1100373233=1, srv1674146292=10, srv1149371336=3, srv258406347=15, srv939556074=28, srv1153515720=5, srv1153246207=4, srv835937613=26, srv51663951=18, srv1125160257=2, srv481017892=17, srv1556299904=8, srv979750212=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2024-04-10 06:56:51,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2024-04-10 06:56:51,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2024-04-10 06:56:51,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2024-04-10 06:56:51,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2024-04-10 06:56:51,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2024-04-10 06:56:51,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2024-04-10 06:56:51,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2024-04-10 06:56:51,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2024-04-10 06:56:51,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2024-04-10 06:56:51,865 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2024-04-10 06:56:51,865 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1030550433:90 , srv1100373233:90 , srv1125160257:90 , srv1149371336:90 , srv1153246207:90 , srv1153515720:90 , srv1240267933:90 , srv1552637476:90 , srv1556299904:90 , srv1655510331:90 , srv1674146292:90 , srv1766907952:90 , srv1898635254:90 , srv1935386394:90 , srv227155055:90 , srv258406347:90 , srv272885343:90 , srv481017892:90 , srv51663951:90 , srv624036704:90 , srv63621446:90 , srv637396533:90 , srv712347512:90 , srv733551302:90 , srv74212547:90 , srv794879293:90 , srv835937613:90 , srv893212801:90 , srv939556074:90 , srv979750212:90 } 2024-04-10 06:56:51,877 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=149 (was 246), ProcessCount=178 (was 180), AvailableMemoryMB=8139 (was 6013) - AvailableMemoryMB LEAK? -