2023-03-23 23:56:12,867 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-03-23 23:56:13,272 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-23 23:56:13,421 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-23 23:56:13,503 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-23 23:56:13,512 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-23 23:56:13,538 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=352, ProcessCount=169, AvailableMemoryMB=6382 2023-03-23 23:56:13,541 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-23 23:56:13,542 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-23 23:56:13,741 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1010336667:84 , srv1012091199:84 , srv1018566201:84 , srv1100376374:84 , srv1130549186:84 , srv1133921980:84 , srv115039080:84 , srv1207932935:84 , srv1226254057:84 , srv1232999617:84 , srv1251045117:84 , srv125514363:84 , srv131425275:84 , srv1314679288:84 , srv1315436783:84 , srv134190301:84 , srv1374031000:84 , srv1381587968:84 , srv1382829056:84 , srv1386247233:84 , srv1414811595:84 , srv1427471851:84 , srv1442920348:84 , srv1444653945:84 , srv1447004705:84 , srv1453475434:84 , srv1460319218:84 , srv1464008093:84 , srv1466042136:84 , srv146711256:84 , srv1473124680:84 , srv1486741689:84 , srv1496921721:84 , srv1505354020:84 , srv1508783935:84 , srv1521314185:84 , srv1538499597:84 , srv1565657339:84 , srv1569878481:84 , srv1579572192:84 , srv161822942:84 , srv1621258251:84 , srv1627650364:84 , srv1638015461:84 , srv1720264745:84 , srv1794980816:84 , srv1803949008:84 , srv181098096:84 , srv1827619236:84 , srv1839546136:84 , srv1864438440:84 , srv1903441321:84 , srv1909490598:84 , srv1946796383:84 , srv2030594277:84 , srv2031639208:84 , srv204225677:84 , srv2119640864:84 , srv213647621:84 , srv214249668:84 , srv247217513:84 , srv279099600:84 , srv296866240:84 , srv30181634:84 , srv314278628:84 , srv316157527:84 , srv326564824:84 , srv336551158:84 , srv340588:84 , srv365211539:84 , srv398095069:84 , srv410804702:84 , srv53120231:84 , srv533718774:84 , srv536416643:84 , srv542437505:84 , srv632381883:84 , srv673779746:84 , srv684068285:84 , srv689436546:84 , srv714252315:84 , srv724776507:84 , srv731815222:84 , srv742285621:84 , srv800198944:84 , srv813184191:84 , srv820296066:84 , srv820908129:84 , srv831886970:84 , srv865004767:84 , srv866946598:84 , srv874772488:84 , srv913135611:84 , srv938431048:84 , srv942891608:84 , srv956401610:84 , srv957358878:84 , srv963081282:84 , srv992324145:84 , srv1245099351:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-03-23 23:56:13,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-23 23:56:13,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1100376374=3, srv161822942=41, srv1720264745=45, srv1496921721=33, srv1453475434=26, srv1018566201=2, srv1460319218=27, srv724776507=82, srv316157527=66, srv134190301=16, srv213647621=59, srv1133921980=5, srv1839546136=50, srv963081282=98, srv1505354020=34, srv1508783935=35, srv1579572192=40, srv115039080=6, srv800198944=85, srv938431048=94, srv340588=69, srv1864438440=51, srv1466042136=29, srv279099600=62, srv1427471851=22, srv125514363=12, srv30181634=64, srv410804702=72, srv1447004705=25, srv1627650364=43, srv957358878=97, srv866946598=91, srv1569878481=39, srv1374031000=17, srv913135611=93, srv1314679288=14, srv865004767=90, srv742285621=84, srv684068285=79, srv326564824=67, srv2119640864=58, srv731815222=83, srv1010336667=0, srv632381883=77, srv1382829056=19, srv874772488=92, srv1621258251=42, srv53120231=73, srv1414811595=21, srv542437505=76, srv1794980816=46, srv533718774=74, srv1444653945=24, srv1315436783=15, srv365211539=70, srv1245099351=10, srv1207932935=7, srv813184191=86, srv1521314185=36, srv820296066=87, srv714252315=81, srv1909490598=53, srv1827619236=49, srv1442920348=23, srv296866240=63, srv1803949008=47, srv336551158=68, srv204225677=57, srv1946796383=54, srv131425275=13, srv992324145=99, srv247217513=61, srv1565657339=38, srv942891608=95, srv1232999617=9, srv1386247233=20, srv1130549186=4, srv1638015461=44, srv1226254057=8, srv673779746=78, srv2030594277=55, srv820908129=88, srv214249668=60, srv1381587968=18, srv1473124680=31, srv181098096=48, srv956401610=96, srv536416643=75, srv1251045117=11, srv398095069=71, srv1464008093=28, srv1486741689=32, srv1903441321=52, srv146711256=30, srv1538499597=37, srv689436546=80, srv1012091199=1, srv831886970=89, srv2031639208=56, srv314278628=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-23 23:56:13,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-23 23:56:13,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-23 23:56:13,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-23 23:56:13,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-23 23:56:13,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-23 23:56:13,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-23 23:56:13,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-23 23:56:13,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-23 23:56:13,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-23 23:56:13,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-23 23:56:13,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-23 23:56:13,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-23 23:56:13,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-23 23:56:13,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-23 23:56:13,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-23 23:56:13,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-23 23:56:13,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-23 23:56:13,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-23 23:56:13,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-23 23:56:13,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-23 23:56:13,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-23 23:56:13,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-23 23:56:13,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-23 23:56:13,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-23 23:56:13,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-23 23:56:13,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-23 23:56:13,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-23 23:56:13,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-23 23:56:13,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-23 23:56:13,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-23 23:56:13,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-23 23:56:13,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-23 23:56:13,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-23 23:56:13,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-23 23:56:13,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-23 23:56:13,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-23 23:56:13,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-23 23:56:13,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-23 23:56:13,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-23 23:56:13,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-23 23:56:13,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-23 23:56:13,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-23 23:56:13,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-23 23:56:13,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-23 23:56:13,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-23 23:56:13,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-23 23:56:13,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-23 23:56:13,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-23 23:56:13,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-23 23:56:13,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-23 23:56:13,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-23 23:56:13,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-23 23:56:13,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-23 23:56:13,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-23 23:56:13,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-23 23:56:13,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-23 23:56:13,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-23 23:56:13,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-23 23:56:13,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-23 23:56:13,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-23 23:56:13,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-23 23:56:13,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-23 23:56:13,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-23 23:56:13,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-23 23:56:13,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-23 23:56:13,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-23 23:56:13,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-23 23:56:13,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-23 23:56:13,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-23 23:56:13,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-23 23:56:13,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-23 23:56:13,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-23 23:56:13,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-23 23:56:13,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-23 23:56:13,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-23 23:56:13,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-23 23:56:13,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-23 23:56:13,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-23 23:56:13,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-23 23:56:13,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-23 23:56:13,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-23 23:56:13,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-23 23:56:13,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-23 23:56:13,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-23 23:56:13,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-23 23:56:13,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-23 23:56:13,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-23 23:56:13,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-23 23:56:13,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-23 23:56:13,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-23 23:56:13,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-23 23:56:13,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-23 23:56:13,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-23 23:56:13,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-23 23:56:13,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-23 23:56:13,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-23 23:56:13,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-23 23:56:13,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-23 23:56:13,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-23 23:56:13,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-23 23:56:13,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-23 23:56:14,207 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-23 23:56:14,207 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-23 23:56:14,209 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-23 23:56:14,673 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 693 ms to try 5000 different iterations. Found a solution that moves 4651 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18325609124626951. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041860342990129425, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05404823883543405, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5167777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041860342990129425, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.179, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24025, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-23 23:56:14,689 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv53120231:80 , srv161822942:81 , srv131425275:82 , srv1521314185:82 , srv1627650364:82 , srv1638015461:82 , srv1130549186:83 , srv2031639208:83 , srv536416643:83 , srv542437505:83 , srv714252315:83 , srv874772488:83 , srv942891608:83 , srv1100376374:84 , srv1315436783:84 , srv1442920348:84 , srv1466042136:84 , srv1486741689:84 , srv1803949008:84 , srv1909490598:84 , srv204225677:84 , srv2119640864:84 , srv340588:84 , srv820296066:84 , srv1226254057:85 , srv1251045117:85 , srv125514363:85 , srv1381587968:85 , srv1386247233:85 , srv1444653945:85 , srv1460319218:85 , srv1464008093:85 , srv181098096:85 , srv1827619236:85 , srv1864438440:85 , srv213647621:85 , srv214249668:85 , srv247217513:85 , srv296866240:85 , srv30181634:85 , srv314278628:85 , srv316157527:85 , srv336551158:85 , srv410804702:85 , srv632381883:85 , srv724776507:85 , srv1505354020:86 , srv1621258251:86 , srv2030594277:86 , srv398095069:86 , srv742285621:86 , srv813184191:86 , srv913135611:86 , srv938431048:86 , srv956401610:86 , srv1012091199:87 , srv115039080:87 , srv1414811595:87 , srv1473124680:87 , srv1508783935:87 , srv1538499597:87 , srv1839546136:87 , srv1903441321:87 , srv1946796383:87 , srv689436546:87 , srv731815222:87 , srv831886970:87 , srv957358878:87 , srv992324145:87 , srv1133921980:88 , srv1207932935:88 , srv1382829056:88 , srv1447004705:88 , srv1569878481:88 , srv1720264745:88 , srv365211539:88 , srv533718774:88 , srv684068285:88 , srv800198944:88 , srv866946598:88 , srv963081282:88 , srv1010336667:89 , srv1232999617:89 , srv1374031000:89 , srv1453475434:89 , srv146711256:89 , srv1794980816:89 , srv279099600:89 , srv326564824:89 , srv865004767:89 , srv1018566201:90 , srv1427471851:90 , srv1496921721:90 , srv673779746:90 , srv134190301:91 , srv1565657339:91 , srv1579572192:92 , srv820908129:93 , srv1314679288:98 , srv1245099351:462 } 2023-03-23 23:56:14,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-23 23:56:14,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1100376374=3, srv161822942=41, srv1720264745=45, srv1496921721=33, srv1453475434=26, srv1018566201=2, srv1460319218=27, srv724776507=82, srv316157527=66, srv134190301=16, srv213647621=59, srv1133921980=5, srv1839546136=50, srv963081282=98, srv1505354020=34, srv1508783935=35, srv1579572192=40, srv115039080=6, srv800198944=85, srv938431048=94, srv340588=69, srv1864438440=51, srv1466042136=29, srv279099600=62, srv1427471851=22, srv125514363=12, srv30181634=64, srv410804702=72, srv1447004705=25, srv1627650364=43, srv957358878=97, srv866946598=91, srv1569878481=39, srv1374031000=17, srv913135611=93, srv1314679288=14, srv865004767=90, srv742285621=84, srv684068285=79, srv326564824=67, srv2119640864=58, srv731815222=83, srv1010336667=0, srv632381883=77, srv1382829056=19, srv874772488=92, srv1621258251=42, srv53120231=73, srv1414811595=21, srv542437505=76, srv1794980816=46, srv533718774=74, srv1444653945=24, srv1315436783=15, srv365211539=70, srv1245099351=10, srv1207932935=7, srv813184191=86, srv1521314185=36, srv820296066=87, srv714252315=81, srv1909490598=53, srv1827619236=49, srv1442920348=23, srv296866240=63, srv1803949008=47, srv336551158=68, srv204225677=57, srv1946796383=54, srv131425275=13, srv992324145=99, srv247217513=61, srv1565657339=38, srv942891608=95, srv1232999617=9, srv1386247233=20, srv1130549186=4, srv1638015461=44, srv1226254057=8, srv673779746=78, srv2030594277=55, srv820908129=88, srv214249668=60, srv1381587968=18, srv1473124680=31, srv181098096=48, srv956401610=96, srv536416643=75, srv1251045117=11, srv398095069=71, srv1464008093=28, srv1486741689=32, srv1903441321=52, srv146711256=30, srv1538499597=37, srv689436546=80, srv1012091199=1, srv831886970=89, srv2031639208=56, srv314278628=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-23 23:56:14,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-23 23:56:14,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-23 23:56:14,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-23 23:56:14,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-23 23:56:14,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-23 23:56:14,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-23 23:56:14,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-23 23:56:14,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-23 23:56:14,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-23 23:56:14,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-23 23:56:14,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-23 23:56:14,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-23 23:56:14,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-23 23:56:14,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-23 23:56:14,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-23 23:56:14,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-23 23:56:14,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-23 23:56:14,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-23 23:56:14,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-23 23:56:14,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-23 23:56:14,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-23 23:56:14,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-23 23:56:14,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-23 23:56:14,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-23 23:56:14,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-23 23:56:14,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-23 23:56:14,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-23 23:56:14,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-23 23:56:14,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-23 23:56:14,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-23 23:56:14,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-23 23:56:14,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-23 23:56:14,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-23 23:56:14,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-23 23:56:14,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-23 23:56:14,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-23 23:56:14,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-23 23:56:14,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-23 23:56:14,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-23 23:56:14,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-23 23:56:14,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-23 23:56:14,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-23 23:56:14,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-23 23:56:14,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-23 23:56:14,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-23 23:56:14,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-23 23:56:14,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-23 23:56:14,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-23 23:56:14,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-23 23:56:14,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-23 23:56:14,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-23 23:56:14,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-23 23:56:14,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-23 23:56:14,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-23 23:56:14,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-23 23:56:14,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-23 23:56:14,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-23 23:56:14,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-23 23:56:14,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-23 23:56:14,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-23 23:56:14,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-23 23:56:14,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-23 23:56:14,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-23 23:56:14,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-23 23:56:14,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-23 23:56:14,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-23 23:56:14,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-23 23:56:14,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-23 23:56:14,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-23 23:56:14,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-23 23:56:14,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-23 23:56:14,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-23 23:56:14,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-23 23:56:14,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-23 23:56:14,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-23 23:56:14,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-23 23:56:14,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-23 23:56:14,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-23 23:56:14,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-23 23:56:14,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-23 23:56:14,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-23 23:56:14,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-23 23:56:14,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-23 23:56:14,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-23 23:56:14,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-23 23:56:14,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-23 23:56:14,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-23 23:56:14,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-23 23:56:14,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-23 23:56:14,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-23 23:56:14,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-23 23:56:14,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-23 23:56:14,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-23 23:56:14,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-23 23:56:14,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-23 23:56:14,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-23 23:56:14,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-23 23:56:14,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-23 23:56:14,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-23 23:56:14,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-23 23:56:14,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-23 23:56:14,943 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041860342990129425, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05404823883543405, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041860342990129425, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.179, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24025, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-23 23:56:14,943 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-23 23:56:14,944 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18322352424820113, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041860342990129425, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05404823883543405, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041860342990129425, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.179, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24025, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-23 23:56:15,307 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 503 ms to try 5000 different iterations. Found a solution that moves 2636 regions; Going from a computed imbalance of 0.18322352424820113 to a new imbalance of 0.015392863790186092. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022827339253266327); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031060564999562248, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29288888888888887, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022827339253266327); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042166666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-23 23:56:15,321 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1314679288:74 , srv820908129:77 , srv1018566201:80 , srv1579572192:80 , srv279099600:80 , srv134190301:81 , srv146711256:81 , srv1720264745:81 , srv865004767:81 , srv1207932935:82 , srv1414811595:82 , srv1427471851:82 , srv1565657339:82 , srv1794980816:82 , srv673779746:82 , srv1010336667:83 , srv1232999617:83 , srv1453475434:83 , srv1496921721:83 , srv326564824:83 , srv533718774:83 , srv800198944:83 , srv115039080:84 , srv1374031000:84 , srv1447004705:84 , srv1508783935:84 , srv1538499597:84 , srv1569878481:84 , srv1946796383:84 , srv365211539:84 , srv684068285:84 , srv813184191:84 , srv866946598:84 , srv1133921980:85 , srv1382829056:85 , srv1473124680:85 , srv1621258251:85 , srv398095069:85 , srv724776507:85 , srv731815222:85 , srv963081282:85 , srv1012091199:86 , srv1251045117:86 , srv1460319218:86 , srv1466042136:86 , srv1505354020:86 , srv181098096:86 , srv1839546136:86 , srv214249668:86 , srv336551158:86 , srv689436546:86 , srv831886970:86 , srv913135611:86 , srv956401610:86 , srv957358878:86 , srv992324145:86 , srv1130549186:87 , srv125514363:87 , srv1315436783:87 , srv1381587968:87 , srv1803949008:87 , srv1827619236:87 , srv1864438440:87 , srv1903441321:87 , srv296866240:87 , srv316157527:87 , srv820296066:87 , srv874772488:87 , srv938431048:87 , srv1100376374:88 , srv1226254057:88 , srv1386247233:88 , srv1444653945:88 , srv1464008093:88 , srv1486741689:88 , srv1521314185:88 , srv2119640864:88 , srv213647621:88 , srv30181634:88 , srv314278628:88 , srv536416643:88 , srv742285621:88 , srv1627650364:89 , srv1909490598:89 , srv2030594277:89 , srv247217513:89 , srv340588:89 , srv542437505:89 , srv632381883:89 , srv714252315:89 , srv131425275:90 , srv1442920348:90 , srv2031639208:90 , srv204225677:90 , srv410804702:90 , srv942891608:90 , srv161822942:91 , srv1638015461:92 , srv53120231:93 , srv1245099351:515 } 2023-03-23 23:56:15,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-23 23:56:15,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1100376374=3, srv161822942=41, srv1720264745=45, srv1496921721=33, srv1453475434=26, srv1018566201=2, srv1460319218=27, srv724776507=82, srv316157527=66, srv134190301=16, srv213647621=59, srv1133921980=5, srv1839546136=50, srv963081282=98, srv1505354020=34, srv1508783935=35, srv1579572192=40, srv115039080=6, srv800198944=85, srv938431048=94, srv340588=69, srv1864438440=51, srv1466042136=29, srv279099600=62, srv1427471851=22, srv125514363=12, srv30181634=64, srv410804702=72, srv1447004705=25, srv1627650364=43, srv957358878=97, srv866946598=91, srv1569878481=39, srv1374031000=17, srv913135611=93, srv1314679288=14, srv865004767=90, srv742285621=84, srv684068285=79, srv326564824=67, srv2119640864=58, srv731815222=83, srv1010336667=0, srv632381883=77, srv1382829056=19, srv874772488=92, srv1621258251=42, srv53120231=73, srv1414811595=21, srv542437505=76, srv1794980816=46, srv533718774=74, srv1444653945=24, srv1315436783=15, srv365211539=70, srv1245099351=10, srv1207932935=7, srv813184191=86, srv1521314185=36, srv820296066=87, srv714252315=81, srv1909490598=53, srv1827619236=49, srv1442920348=23, srv296866240=63, srv1803949008=47, srv336551158=68, srv204225677=57, srv1946796383=54, srv131425275=13, srv992324145=99, srv247217513=61, srv1565657339=38, srv942891608=95, srv1232999617=9, srv1386247233=20, srv1130549186=4, srv1638015461=44, srv1226254057=8, srv673779746=78, srv2030594277=55, srv820908129=88, srv214249668=60, srv1381587968=18, srv1473124680=31, srv181098096=48, srv956401610=96, srv536416643=75, srv1251045117=11, srv398095069=71, srv1464008093=28, srv1486741689=32, srv1903441321=52, srv146711256=30, srv1538499597=37, srv689436546=80, srv1012091199=1, srv831886970=89, srv2031639208=56, srv314278628=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-23 23:56:15,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-23 23:56:15,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-23 23:56:15,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-23 23:56:15,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-23 23:56:15,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-23 23:56:15,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-23 23:56:15,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-23 23:56:15,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-23 23:56:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-23 23:56:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-23 23:56:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-23 23:56:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-23 23:56:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-23 23:56:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-23 23:56:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-23 23:56:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-23 23:56:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-23 23:56:15,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-23 23:56:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-23 23:56:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-23 23:56:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-23 23:56:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-23 23:56:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-23 23:56:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-23 23:56:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-23 23:56:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-23 23:56:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-23 23:56:15,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-23 23:56:15,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-23 23:56:15,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-23 23:56:15,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-23 23:56:15,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-23 23:56:15,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-23 23:56:15,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-23 23:56:15,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-23 23:56:15,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-23 23:56:15,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-23 23:56:15,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-23 23:56:15,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-23 23:56:15,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-23 23:56:15,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-23 23:56:15,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-23 23:56:15,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-23 23:56:15,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-23 23:56:15,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-23 23:56:15,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-23 23:56:15,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-23 23:56:15,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-23 23:56:15,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-23 23:56:15,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-23 23:56:15,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-23 23:56:15,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-23 23:56:15,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-23 23:56:15,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-23 23:56:15,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-23 23:56:15,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-23 23:56:15,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-23 23:56:15,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-23 23:56:15,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-23 23:56:15,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-23 23:56:15,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-23 23:56:15,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-23 23:56:15,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-23 23:56:15,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-23 23:56:15,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-23 23:56:15,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-23 23:56:15,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-23 23:56:15,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-23 23:56:15,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-23 23:56:15,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-23 23:56:15,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-23 23:56:15,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-23 23:56:15,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-23 23:56:15,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-23 23:56:15,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-23 23:56:15,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-23 23:56:15,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-23 23:56:15,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-23 23:56:15,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-23 23:56:15,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-23 23:56:15,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-23 23:56:15,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-23 23:56:15,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-23 23:56:15,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-23 23:56:15,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-23 23:56:15,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-23 23:56:15,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-23 23:56:15,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-23 23:56:15,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-23 23:56:15,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-23 23:56:15,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-23 23:56:15,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-23 23:56:15,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-23 23:56:15,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-23 23:56:15,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-23 23:56:15,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-23 23:56:15,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-23 23:56:15,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-23 23:56:15,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-23 23:56:15,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-23 23:56:15,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-23 23:56:15,500 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022827339253266327); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031060564999562248, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022827339253266327); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042166666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-23 23:56:15,500 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-23 23:56:15,500 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015374406123682476, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022827339253266327); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031060564999562248, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022827339253266327); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042166666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-23 23:56:15,877 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 497 ms to try 5000 different iterations. Found a solution that moves 963 regions; Going from a computed imbalance of 0.015374406123682476 to a new imbalance of 4.497213550277273E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00510031553982802); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006285393610547089); MoveCostFunction : (multiplier=7.0, imbalance=0.107, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00510031553982802); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-23 23:56:15,879 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1442920348:83 , srv1638015461:83 , srv1903441321:83 , srv204225677:83 , srv410804702:83 , srv1374031000:84 , srv1386247233:84 , srv1444653945:84 , srv1453475434:84 , srv1839546136:84 , srv1909490598:84 , srv2030594277:84 , srv2031639208:84 , srv247217513:84 , srv340588:84 , srv53120231:84 , srv632381883:84 , srv742285621:84 , srv938431048:84 , srv942891608:84 , srv963081282:84 , srv992324145:84 , srv1010336667:85 , srv1012091199:85 , srv1100376374:85 , srv1133921980:85 , srv1226254057:85 , srv1382829056:85 , srv1464008093:85 , srv1496921721:85 , srv1505354020:85 , srv1565657339:85 , srv1569878481:85 , srv1579572192:85 , srv1864438440:85 , srv30181634:85 , srv314278628:85 , srv316157527:85 , srv326564824:85 , srv542437505:85 , srv689436546:85 , srv714252315:85 , srv866946598:85 , srv913135611:85 , srv956401610:85 , srv957358878:85 , srv115039080:86 , srv1232999617:86 , srv125514363:86 , srv131425275:86 , srv1314679288:86 , srv134190301:86 , srv1381587968:86 , srv1427471851:86 , srv1447004705:86 , srv1473124680:86 , srv1508783935:86 , srv1538499597:86 , srv161822942:86 , srv1621258251:86 , srv1627650364:86 , srv1794980816:86 , srv1803949008:86 , srv181098096:86 , srv1827619236:86 , srv2119640864:86 , srv213647621:86 , srv296866240:86 , srv365211539:86 , srv533718774:86 , srv673779746:86 , srv684068285:86 , srv731815222:86 , srv800198944:86 , srv820296066:86 , srv831886970:86 , srv1018566201:87 , srv1130549186:87 , srv1251045117:87 , srv1315436783:87 , srv1460319218:87 , srv146711256:87 , srv1486741689:87 , srv1521314185:87 , srv1946796383:87 , srv214249668:87 , srv336551158:87 , srv398095069:87 , srv536416643:87 , srv813184191:87 , srv865004767:87 , srv874772488:87 , srv1414811595:88 , srv1466042136:88 , srv279099600:88 , srv724776507:88 , srv820908129:88 , srv1720264745:89 , srv1207932935:90 , srv1245099351:526 } 2023-03-23 23:56:15,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-23 23:56:15,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1100376374=3, srv161822942=41, srv1720264745=45, srv1496921721=33, srv1453475434=26, srv1018566201=2, srv1460319218=27, srv724776507=82, srv316157527=66, srv134190301=16, srv213647621=59, srv1133921980=5, srv1839546136=50, srv963081282=98, srv1505354020=34, srv1508783935=35, srv1579572192=40, srv115039080=6, srv800198944=85, srv938431048=94, srv340588=69, srv1864438440=51, srv1466042136=29, srv279099600=62, srv1427471851=22, srv125514363=12, srv30181634=64, srv410804702=72, srv1447004705=25, srv1627650364=43, srv957358878=97, srv866946598=91, srv1569878481=39, srv1374031000=17, srv913135611=93, srv1314679288=14, srv865004767=90, srv742285621=84, srv684068285=79, srv326564824=67, srv2119640864=58, srv731815222=83, srv1010336667=0, srv632381883=77, srv1382829056=19, srv874772488=92, srv1621258251=42, srv53120231=73, srv1414811595=21, srv542437505=76, srv1794980816=46, srv533718774=74, srv1444653945=24, srv1315436783=15, srv365211539=70, srv1245099351=10, srv1207932935=7, srv813184191=86, srv1521314185=36, srv820296066=87, srv714252315=81, srv1909490598=53, srv1827619236=49, srv1442920348=23, srv296866240=63, srv1803949008=47, srv336551158=68, srv204225677=57, srv1946796383=54, srv131425275=13, srv992324145=99, srv247217513=61, srv1565657339=38, srv942891608=95, srv1232999617=9, srv1386247233=20, srv1130549186=4, srv1638015461=44, srv1226254057=8, srv673779746=78, srv2030594277=55, srv820908129=88, srv214249668=60, srv1381587968=18, srv1473124680=31, srv181098096=48, srv956401610=96, srv536416643=75, srv1251045117=11, srv398095069=71, srv1464008093=28, srv1486741689=32, srv1903441321=52, srv146711256=30, srv1538499597=37, srv689436546=80, srv1012091199=1, srv831886970=89, srv2031639208=56, srv314278628=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-23 23:56:15,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-23 23:56:15,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-23 23:56:15,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-23 23:56:15,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-23 23:56:15,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-23 23:56:15,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-23 23:56:15,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-23 23:56:15,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-23 23:56:15,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-23 23:56:15,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-23 23:56:15,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-23 23:56:15,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-23 23:56:15,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-23 23:56:15,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-23 23:56:15,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-23 23:56:15,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-23 23:56:15,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-23 23:56:15,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-23 23:56:15,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-23 23:56:15,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-23 23:56:15,910 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-23 23:56:15,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-23 23:56:15,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-23 23:56:15,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-23 23:56:15,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-23 23:56:15,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-23 23:56:15,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-23 23:56:15,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-23 23:56:15,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-23 23:56:15,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-23 23:56:15,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-23 23:56:15,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-23 23:56:15,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-23 23:56:15,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-23 23:56:15,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-23 23:56:15,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-23 23:56:15,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-23 23:56:15,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-23 23:56:15,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-23 23:56:15,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-23 23:56:15,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-23 23:56:15,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-23 23:56:15,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-23 23:56:15,912 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-23 23:56:15,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-23 23:56:15,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-23 23:56:15,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-23 23:56:15,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-23 23:56:15,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-23 23:56:15,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-23 23:56:15,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-23 23:56:15,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-23 23:56:15,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-23 23:56:15,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-23 23:56:15,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-23 23:56:15,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-23 23:56:15,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-23 23:56:15,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-23 23:56:15,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-23 23:56:15,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-23 23:56:15,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-23 23:56:15,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-23 23:56:15,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-23 23:56:15,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-23 23:56:15,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-23 23:56:15,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-23 23:56:15,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-23 23:56:15,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-23 23:56:15,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-23 23:56:15,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-23 23:56:15,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-23 23:56:15,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-23 23:56:15,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-23 23:56:15,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-23 23:56:15,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-23 23:56:15,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-23 23:56:15,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-23 23:56:15,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-23 23:56:15,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-23 23:56:15,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-23 23:56:15,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-23 23:56:15,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-23 23:56:15,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-23 23:56:15,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-23 23:56:15,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-23 23:56:15,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-23 23:56:15,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-23 23:56:15,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-23 23:56:15,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-23 23:56:15,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-23 23:56:15,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-23 23:56:15,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-23 23:56:15,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-23 23:56:15,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-23 23:56:15,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-23 23:56:15,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-23 23:56:15,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-23 23:56:15,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-23 23:56:15,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-23 23:56:15,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-23 23:56:15,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-23 23:56:16,045 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00510031553982802); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006285393610547089); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00510031553982802); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-23 23:56:16,046 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-23 23:56:16,046 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.4297828490519976E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00510031553982802); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006285393610547089); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00510031553982802); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-23 23:56:16,447 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 514 ms to try 5000 different iterations. Found a solution that moves 230 regions; Going from a computed imbalance of 4.4297828490519976E-4 to a new imbalance of 4.203091265328697E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.025555555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-23 23:56:16,448 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1207932935:82 , srv1382829056:83 , srv1486741689:83 , srv1565657339:83 , srv2030594277:83 , srv213647621:83 , srv831886970:83 , srv1012091199:84 , srv1133921980:84 , srv1226254057:84 , srv1232999617:84 , srv1251045117:84 , srv125514363:84 , srv131425275:84 , srv1314679288:84 , srv1315436783:84 , srv134190301:84 , srv1381587968:84 , srv1427471851:84 , srv1447004705:84 , srv1464008093:84 , srv1473124680:84 , srv1496921721:84 , srv161822942:84 , srv1794980816:84 , srv1803949008:84 , srv1946796383:84 , srv2119640864:84 , srv214249668:84 , srv247217513:84 , srv279099600:84 , srv296866240:84 , srv30181634:84 , srv314278628:84 , srv336551158:84 , srv365211539:84 , srv398095069:84 , srv410804702:84 , srv533718774:84 , srv632381883:84 , srv673779746:84 , srv684068285:84 , srv689436546:84 , srv724776507:84 , srv731815222:84 , srv742285621:84 , srv820908129:84 , srv913135611:84 , srv957358878:84 , srv1010336667:85 , srv1100376374:85 , srv1130549186:85 , srv115039080:85 , srv1374031000:85 , srv1386247233:85 , srv1414811595:85 , srv1442920348:85 , srv1444653945:85 , srv1460319218:85 , srv1466042136:85 , srv146711256:85 , srv1505354020:85 , srv1508783935:85 , srv1521314185:85 , srv1538499597:85 , srv1569878481:85 , srv1579572192:85 , srv1621258251:85 , srv1627650364:85 , srv1638015461:85 , srv1720264745:85 , srv181098096:85 , srv1827619236:85 , srv1839546136:85 , srv1864438440:85 , srv1903441321:85 , srv1909490598:85 , srv2031639208:85 , srv204225677:85 , srv316157527:85 , srv326564824:85 , srv340588:85 , srv53120231:85 , srv536416643:85 , srv542437505:85 , srv714252315:85 , srv800198944:85 , srv813184191:85 , srv820296066:85 , srv865004767:85 , srv866946598:85 , srv938431048:85 , srv942891608:85 , srv963081282:85 , srv992324145:85 , srv1018566201:86 , srv1453475434:86 , srv874772488:86 , srv956401610:86 , srv1245099351:638 } 2023-03-23 23:56:16,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-23 23:56:16,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1100376374=3, srv161822942=41, srv1720264745=45, srv1496921721=33, srv1453475434=26, srv1018566201=2, srv1460319218=27, srv724776507=82, srv316157527=66, srv134190301=16, srv213647621=59, srv1133921980=5, srv1839546136=50, srv963081282=98, srv1505354020=34, srv1508783935=35, srv1579572192=40, srv115039080=6, srv800198944=85, srv938431048=94, srv340588=69, srv1864438440=51, srv1466042136=29, srv279099600=62, srv1427471851=22, srv125514363=12, srv30181634=64, srv410804702=72, srv1447004705=25, srv1627650364=43, srv957358878=97, srv866946598=91, srv1569878481=39, srv1374031000=17, srv913135611=93, srv1314679288=14, srv865004767=90, srv742285621=84, srv684068285=79, srv326564824=67, srv2119640864=58, srv731815222=83, srv1010336667=0, srv632381883=77, srv1382829056=19, srv874772488=92, srv1621258251=42, srv53120231=73, srv1414811595=21, srv542437505=76, srv1794980816=46, srv533718774=74, srv1444653945=24, srv1315436783=15, srv365211539=70, srv1245099351=10, srv1207932935=7, srv813184191=86, srv1521314185=36, srv820296066=87, srv714252315=81, srv1909490598=53, srv1827619236=49, srv1442920348=23, srv296866240=63, srv1803949008=47, srv336551158=68, srv204225677=57, srv1946796383=54, srv131425275=13, srv992324145=99, srv247217513=61, srv1565657339=38, srv942891608=95, srv1232999617=9, srv1386247233=20, srv1130549186=4, srv1638015461=44, srv1226254057=8, srv673779746=78, srv2030594277=55, srv820908129=88, srv214249668=60, srv1381587968=18, srv1473124680=31, srv181098096=48, srv956401610=96, srv536416643=75, srv1251045117=11, srv398095069=71, srv1464008093=28, srv1486741689=32, srv1903441321=52, srv146711256=30, srv1538499597=37, srv689436546=80, srv1012091199=1, srv831886970=89, srv2031639208=56, srv314278628=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-23 23:56:16,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-23 23:56:16,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-23 23:56:16,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-23 23:56:16,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-23 23:56:16,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-23 23:56:16,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-23 23:56:16,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-23 23:56:16,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-23 23:56:16,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-23 23:56:16,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-23 23:56:16,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-23 23:56:16,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-23 23:56:16,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-23 23:56:16,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-23 23:56:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-23 23:56:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-23 23:56:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-23 23:56:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-23 23:56:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-23 23:56:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-23 23:56:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-23 23:56:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-23 23:56:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-23 23:56:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-23 23:56:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-23 23:56:16,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-23 23:56:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-23 23:56:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-23 23:56:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-23 23:56:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-23 23:56:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-23 23:56:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-23 23:56:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-23 23:56:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-23 23:56:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-23 23:56:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-23 23:56:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-23 23:56:16,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-23 23:56:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-23 23:56:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-23 23:56:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-23 23:56:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-23 23:56:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-23 23:56:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-23 23:56:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-23 23:56:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-23 23:56:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-23 23:56:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-23 23:56:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-23 23:56:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-23 23:56:16,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-23 23:56:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-23 23:56:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-23 23:56:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-23 23:56:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-23 23:56:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-23 23:56:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-23 23:56:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-23 23:56:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-23 23:56:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-23 23:56:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-23 23:56:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-23 23:56:16,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-23 23:56:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-23 23:56:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-23 23:56:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-23 23:56:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-23 23:56:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-23 23:56:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-23 23:56:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-23 23:56:16,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-23 23:56:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-23 23:56:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-23 23:56:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-23 23:56:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-23 23:56:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-23 23:56:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-23 23:56:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-23 23:56:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-23 23:56:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-23 23:56:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-23 23:56:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-23 23:56:16,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-23 23:56:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-23 23:56:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-23 23:56:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-23 23:56:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-23 23:56:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-23 23:56:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-23 23:56:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-23 23:56:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-23 23:56:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-23 23:56:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-23 23:56:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-23 23:56:16,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-23 23:56:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-23 23:56:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-23 23:56:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-23 23:56:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-23 23:56:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-23 23:56:16,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-23 23:56:16,625 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-23 23:56:16,625 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-23 23:56:16,626 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.042041823149949E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-23 23:56:16,973 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 482 ms to try 5000 different iterations. Found a solution that moves 25 regions; Going from a computed imbalance of 4.042041823149949E-5 to a new imbalance of 6.919122795330589E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.002777777777777778); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-23 23:56:16,974 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1018566201:83 , srv1460319218:83 , srv1466042136:83 , srv1720264745:83 , srv1827619236:83 , srv536416643:83 , srv874772488:83 , srv956401610:83 , srv1010336667:84 , srv1012091199:84 , srv1100376374:84 , srv1130549186:84 , srv1133921980:84 , srv115039080:84 , srv1207932935:84 , srv1226254057:84 , srv1232999617:84 , srv1251045117:84 , srv125514363:84 , srv131425275:84 , srv1314679288:84 , srv1315436783:84 , srv134190301:84 , srv1374031000:84 , srv1381587968:84 , srv1386247233:84 , srv1414811595:84 , srv1427471851:84 , srv1442920348:84 , srv1444653945:84 , srv1447004705:84 , srv1453475434:84 , srv1464008093:84 , srv146711256:84 , srv1473124680:84 , srv1486741689:84 , srv1496921721:84 , srv1505354020:84 , srv1508783935:84 , srv1521314185:84 , srv1538499597:84 , srv1569878481:84 , srv1579572192:84 , srv161822942:84 , srv1621258251:84 , srv1627650364:84 , srv1638015461:84 , srv181098096:84 , srv1839546136:84 , srv1864438440:84 , srv1903441321:84 , srv1909490598:84 , srv1946796383:84 , srv2030594277:84 , srv2031639208:84 , srv204225677:84 , srv2119640864:84 , srv213647621:84 , srv214249668:84 , srv247217513:84 , srv296866240:84 , srv30181634:84 , srv314278628:84 , srv316157527:84 , srv326564824:84 , srv336551158:84 , srv340588:84 , srv365211539:84 , srv398095069:84 , srv410804702:84 , srv53120231:84 , srv542437505:84 , srv632381883:84 , srv673779746:84 , srv684068285:84 , srv689436546:84 , srv714252315:84 , srv724776507:84 , srv731815222:84 , srv742285621:84 , srv800198944:84 , srv813184191:84 , srv820296066:84 , srv820908129:84 , srv831886970:84 , srv865004767:84 , srv866946598:84 , srv938431048:84 , srv942891608:84 , srv957358878:84 , srv963081282:84 , srv992324145:84 , srv1382829056:85 , srv1565657339:85 , srv1794980816:85 , srv1803949008:85 , srv279099600:85 , srv533718774:85 , srv913135611:85 , srv1245099351:685 } 2023-03-23 23:56:16,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-23 23:56:16,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1100376374=3, srv161822942=41, srv1720264745=45, srv1496921721=33, srv1453475434=26, srv1018566201=2, srv1460319218=27, srv724776507=82, srv316157527=66, srv134190301=16, srv213647621=59, srv1133921980=5, srv1839546136=50, srv963081282=98, srv1505354020=34, srv1508783935=35, srv1579572192=40, srv115039080=6, srv800198944=85, srv938431048=94, srv340588=69, srv1864438440=51, srv1466042136=29, srv279099600=62, srv1427471851=22, srv125514363=12, srv30181634=64, srv410804702=72, srv1447004705=25, srv1627650364=43, srv957358878=97, srv866946598=91, srv1569878481=39, srv1374031000=17, srv913135611=93, srv1314679288=14, srv865004767=90, srv742285621=84, srv684068285=79, srv326564824=67, srv2119640864=58, srv731815222=83, srv1010336667=0, srv632381883=77, srv1382829056=19, srv874772488=92, srv1621258251=42, srv53120231=73, srv1414811595=21, srv542437505=76, srv1794980816=46, srv533718774=74, srv1444653945=24, srv1315436783=15, srv365211539=70, srv1245099351=10, srv1207932935=7, srv813184191=86, srv1521314185=36, srv820296066=87, srv714252315=81, srv1909490598=53, srv1827619236=49, srv1442920348=23, srv296866240=63, srv1803949008=47, srv336551158=68, srv204225677=57, srv1946796383=54, srv131425275=13, srv992324145=99, srv247217513=61, srv1565657339=38, srv942891608=95, srv1232999617=9, srv1386247233=20, srv1130549186=4, srv1638015461=44, srv1226254057=8, srv673779746=78, srv2030594277=55, srv820908129=88, srv214249668=60, srv1381587968=18, srv1473124680=31, srv181098096=48, srv956401610=96, srv536416643=75, srv1251045117=11, srv398095069=71, srv1464008093=28, srv1486741689=32, srv1903441321=52, srv146711256=30, srv1538499597=37, srv689436546=80, srv1012091199=1, srv831886970=89, srv2031639208=56, srv314278628=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-23 23:56:16,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-23 23:56:16,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-23 23:56:16,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-23 23:56:16,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-23 23:56:16,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-23 23:56:16,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-23 23:56:16,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-23 23:56:16,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-23 23:56:16,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-23 23:56:16,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-23 23:56:16,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-23 23:56:16,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-23 23:56:16,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-23 23:56:16,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-23 23:56:16,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-23 23:56:16,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-23 23:56:16,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-23 23:56:16,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-23 23:56:16,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-23 23:56:16,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-23 23:56:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-23 23:56:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-23 23:56:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-23 23:56:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-23 23:56:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-23 23:56:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-23 23:56:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-23 23:56:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-23 23:56:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-23 23:56:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-23 23:56:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-23 23:56:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-23 23:56:16,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-23 23:56:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-23 23:56:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-23 23:56:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-23 23:56:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-23 23:56:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-23 23:56:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-23 23:56:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-23 23:56:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-23 23:56:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-23 23:56:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-23 23:56:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-23 23:56:17,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-23 23:56:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-23 23:56:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-23 23:56:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-23 23:56:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-23 23:56:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-23 23:56:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-23 23:56:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-23 23:56:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-23 23:56:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-23 23:56:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-23 23:56:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-23 23:56:17,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-23 23:56:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-23 23:56:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-23 23:56:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-23 23:56:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-23 23:56:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-23 23:56:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-23 23:56:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-23 23:56:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-23 23:56:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-23 23:56:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-23 23:56:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-23 23:56:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-23 23:56:17,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-23 23:56:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-23 23:56:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-23 23:56:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-23 23:56:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-23 23:56:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-23 23:56:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-23 23:56:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-23 23:56:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-23 23:56:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-23 23:56:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-23 23:56:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-23 23:56:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-23 23:56:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-23 23:56:17,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-23 23:56:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-23 23:56:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-23 23:56:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-23 23:56:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-23 23:56:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-23 23:56:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-23 23:56:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-23 23:56:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-23 23:56:17,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-23 23:56:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-23 23:56:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-23 23:56:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-23 23:56:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-23 23:56:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-23 23:56:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-23 23:56:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-23 23:56:17,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-23 23:56:17,129 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-23 23:56:17,130 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1018566201:83 , srv1460319218:83 , srv1466042136:83 , srv1720264745:83 , srv1827619236:83 , srv536416643:83 , srv874772488:83 , srv956401610:83 , srv1010336667:84 , srv1012091199:84 , srv1100376374:84 , srv1130549186:84 , srv1133921980:84 , srv115039080:84 , srv1207932935:84 , srv1226254057:84 , srv1232999617:84 , srv1251045117:84 , srv125514363:84 , srv131425275:84 , srv1314679288:84 , srv1315436783:84 , srv134190301:84 , srv1374031000:84 , srv1381587968:84 , srv1386247233:84 , srv1414811595:84 , srv1427471851:84 , srv1442920348:84 , srv1444653945:84 , srv1447004705:84 , srv1453475434:84 , srv1464008093:84 , srv146711256:84 , srv1473124680:84 , srv1486741689:84 , srv1496921721:84 , srv1505354020:84 , srv1508783935:84 , srv1521314185:84 , srv1538499597:84 , srv1569878481:84 , srv1579572192:84 , srv161822942:84 , srv1621258251:84 , srv1627650364:84 , srv1638015461:84 , srv181098096:84 , srv1839546136:84 , srv1864438440:84 , srv1903441321:84 , srv1909490598:84 , srv1946796383:84 , srv2030594277:84 , srv2031639208:84 , srv204225677:84 , srv2119640864:84 , srv213647621:84 , srv214249668:84 , srv247217513:84 , srv296866240:84 , srv30181634:84 , srv314278628:84 , srv316157527:84 , srv326564824:84 , srv336551158:84 , srv340588:84 , srv365211539:84 , srv398095069:84 , srv410804702:84 , srv53120231:84 , srv542437505:84 , srv632381883:84 , srv673779746:84 , srv684068285:84 , srv689436546:84 , srv714252315:84 , srv724776507:84 , srv731815222:84 , srv742285621:84 , srv800198944:84 , srv813184191:84 , srv820296066:84 , srv820908129:84 , srv831886970:84 , srv865004767:84 , srv866946598:84 , srv938431048:84 , srv942891608:84 , srv957358878:84 , srv963081282:84 , srv992324145:84 , srv1382829056:85 , srv1565657339:85 , srv1794980816:85 , srv1803949008:85 , srv279099600:85 , srv533718774:85 , srv913135611:85 , srv1245099351:685 } 2023-03-23 23:56:17,159 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=364 (was 352) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=5899 (was 6382) 2023-03-23 23:56:17,169 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=364, ProcessCount=169, AvailableMemoryMB=5898 2023-03-23 23:56:17,169 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-23 23:56:17,170 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-03-23 23:56:17,182 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1029356041:84 , srv1062045888:84 , srv1301894687:84 , srv1394837576:84 , srv1400013246:84 , srv1463217144:84 , srv1685360848:84 , srv1701107205:84 , srv1706920049:84 , srv1819011204:84 , srv1827174453:84 , srv184331251:84 , srv1850361474:84 , srv1883669395:84 , srv1922045063:84 , srv1967926708:84 , srv1978075281:84 , srv2038506084:84 , srv2045575602:84 , srv2112229462:84 , srv268185666:84 , srv433087993:84 , srv47081379:84 , srv56726214:84 , srv571485692:84 , srv571535299:84 , srv78544359:84 , srv793439029:84 , srv964407757:84 , srv6504793:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-03-23 23:56:17,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-23 23:56:17,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv47081379=22, srv1967926708=15, srv78544359=27, srv1922045063=14, srv1706920049=8, srv56726214=23, srv1883669395=13, srv793439029=28, srv1400013246=4, srv1701107205=7, srv1850361474=12, srv6504793=26, srv1819011204=9, srv1394837576=3, srv2038506084=17, srv268185666=20, srv1463217144=5, srv1062045888=1, srv571535299=25, srv964407757=29, srv1685360848=6, srv1978075281=16, srv1827174453=10, srv571485692=24, srv433087993=21, srv1029356041=0, srv184331251=11, srv1301894687=2, srv2045575602=18, srv2112229462=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-23 23:56:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-23 23:56:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-23 23:56:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-23 23:56:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-23 23:56:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-23 23:56:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-23 23:56:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-23 23:56:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-23 23:56:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-23 23:56:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-23 23:56:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-23 23:56:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-23 23:56:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-23 23:56:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-23 23:56:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-23 23:56:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-23 23:56:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-23 23:56:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-23 23:56:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-23 23:56:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-23 23:56:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-23 23:56:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-23 23:56:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-23 23:56:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-23 23:56:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-23 23:56:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-23 23:56:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-23 23:56:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-23 23:56:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-23 23:56:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-23 23:56:17,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-03-23 23:56:17,224 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0766194364652104, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-23 23:56:17,225 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912434468606728, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0766194364652104, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-03-23 23:58:17,204 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5826242 different iterations. Found a solution that moves 1846 regions; Going from a computed imbalance of 0.9912434468606728 to a new imbalance of 4.308656090753194E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6837037037037037, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-23 23:58:17,212 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1029356041:90 , srv1062045888:90 , srv1301894687:90 , srv1394837576:90 , srv1400013246:90 , srv1463217144:90 , srv1685360848:90 , srv1701107205:90 , srv1706920049:90 , srv1819011204:90 , srv1827174453:90 , srv184331251:90 , srv1850361474:90 , srv1883669395:90 , srv1922045063:90 , srv1967926708:90 , srv1978075281:90 , srv2038506084:90 , srv2045575602:90 , srv2112229462:90 , srv268185666:90 , srv433087993:90 , srv47081379:90 , srv56726214:90 , srv571485692:90 , srv571535299:90 , srv6504793:90 , srv78544359:90 , srv793439029:90 , srv964407757:90 } 2023-03-23 23:58:17,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-23 23:58:17,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv47081379=22, srv1967926708=15, srv78544359=27, srv1922045063=14, srv1706920049=8, srv56726214=23, srv1883669395=13, srv793439029=28, srv1400013246=4, srv1701107205=7, srv1850361474=12, srv6504793=26, srv1819011204=9, srv1394837576=3, srv2038506084=17, srv268185666=20, srv1463217144=5, srv1062045888=1, srv571535299=25, srv964407757=29, srv1685360848=6, srv1978075281=16, srv1827174453=10, srv571485692=24, srv433087993=21, srv1029356041=0, srv184331251=11, srv1301894687=2, srv2045575602=18, srv2112229462=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-23 23:58:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-23 23:58:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-23 23:58:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-23 23:58:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-23 23:58:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-23 23:58:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-23 23:58:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-23 23:58:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-23 23:58:17,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-23 23:58:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-23 23:58:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-23 23:58:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-23 23:58:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-23 23:58:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-23 23:58:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-23 23:58:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-23 23:58:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-23 23:58:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-23 23:58:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-23 23:58:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-23 23:58:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-23 23:58:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-23 23:58:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-23 23:58:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-23 23:58:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-23 23:58:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-23 23:58:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-23 23:58:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-23 23:58:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-23 23:58:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-23 23:58:17,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-03-23 23:58:17,230 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-23 23:58:17,230 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1029356041:90 , srv1062045888:90 , srv1301894687:90 , srv1394837576:90 , srv1400013246:90 , srv1463217144:90 , srv1685360848:90 , srv1701107205:90 , srv1706920049:90 , srv1819011204:90 , srv1827174453:90 , srv184331251:90 , srv1850361474:90 , srv1883669395:90 , srv1922045063:90 , srv1967926708:90 , srv1978075281:90 , srv2038506084:90 , srv2045575602:90 , srv2112229462:90 , srv268185666:90 , srv433087993:90 , srv47081379:90 , srv56726214:90 , srv571485692:90 , srv571535299:90 , srv6504793:90 , srv78544359:90 , srv793439029:90 , srv964407757:90 } 2023-03-23 23:58:17,243 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=189 (was 364), ProcessCount=169 (was 169), AvailableMemoryMB=5869 (was 5898)