2023-02-17 09:53:01,110 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-02-17 09:53:01,411 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-17 09:53:01,519 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-17 09:53:01,569 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-17 09:53:01,576 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-17 09:53:01,587 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=379, ProcessCount=174, AvailableMemoryMB=3808 2023-02-17 09:53:01,589 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-17 09:53:01,589 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-17 09:53:01,744 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1005373423:84 , srv1018458968:84 , srv1032248280:84 , srv1043092270:84 , srv1046736216:84 , srv1052727655:84 , srv1062181387:84 , srv1078158888:84 , srv110406127:84 , srv1129960133:84 , srv1151037205:84 , srv1177089415:84 , srv1185043876:84 , srv1193092678:84 , srv1195959051:84 , srv1216700836:84 , srv1223951671:84 , srv1262079392:84 , srv1310228683:84 , srv1317620640:84 , srv1326989513:84 , srv1335954299:84 , srv1366317408:84 , srv1412421884:84 , srv1431037828:84 , srv1432153063:84 , srv1447918517:84 , srv1466361118:84 , srv1519022572:84 , srv1543904210:84 , srv1618933697:84 , srv1625769908:84 , srv1632620530:84 , srv1657001475:84 , srv1691631250:84 , srv1728722955:84 , srv1743992913:84 , srv1803779942:84 , srv1828591227:84 , srv185769862:84 , srv1879329479:84 , srv1880109519:84 , srv1886711737:84 , srv1905967899:84 , srv192262889:84 , srv1923985146:84 , srv1948656322:84 , srv1950507900:84 , srv1962166509:84 , srv1963311855:84 , srv1964988541:84 , srv1995569149:84 , srv2029675636:84 , srv2048131359:84 , srv2069744653:84 , srv2088283381:84 , srv208990820:84 , srv2133296283:84 , srv2133615522:84 , srv22586836:84 , srv245066002:84 , srv280870525:84 , srv297717738:84 , srv306041533:84 , srv339249551:84 , srv341645461:84 , srv347007415:84 , srv367872298:84 , srv402330108:84 , srv422237790:84 , srv467101838:84 , srv502454678:84 , srv508607174:84 , srv528327842:84 , srv615912247:84 , srv628968785:84 , srv644993989:84 , srv651845654:84 , srv689977312:84 , srv697891374:84 , srv702459095:84 , srv745455391:84 , srv755603190:84 , srv772403154:84 , srv791082659:84 , srv851034220:84 , srv854372046:84 , srv8684563:84 , srv86983396:84 , srv91387693:84 , srv91936946:84 , srv924022310:84 , srv933614630:84 , srv935813467:84 , srv943154066:84 , srv949362644:84 , srv954004393:84 , srv987089988:84 , srv987425846:84 , srv120557424:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-02-17 09:53:01,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-17 09:53:01,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1262079392=18, srv528327842=74, srv1447918517=27, srv1129960133=9, srv1880109519=42, srv1964988541=51, srv1963311855=50, srv1962166509=49, srv502454678=72, srv1886711737=43, srv245066002=61, srv1195959051=14, srv1317620640=20, srv628968785=76, srv1657001475=34, srv1948656322=47, srv1005373423=0, srv1193092678=13, srv1691631250=35, srv702459095=81, srv755603190=83, srv1879329479=41, srv1043092270=3, srv508607174=73, srv943154066=95, srv1466361118=28, srv341645461=66, srv467101838=71, srv924022310=92, srv1078158888=7, srv367872298=68, srv1046736216=4, srv745455391=82, srv402330108=69, srv644993989=77, srv1032248280=2, srv91387693=90, srv2048131359=54, srv2133296283=58, srv791082659=85, srv1223951671=17, srv1519022572=29, srv1743992913=37, srv949362644=96, srv1412421884=24, srv120557424=15, srv1923985146=46, srv1326989513=21, srv192262889=45, srv280870525=62, srv22586836=60, srv1310228683=19, srv339249551=65, srv1543904210=30, srv208990820=57, srv933614630=93, srv1828591227=39, srv987089988=98, srv422237790=70, srv851034220=86, srv1728722955=36, srv1216700836=16, srv2133615522=59, srv1950507900=48, srv2029675636=53, srv651845654=78, srv1335954299=22, srv1625769908=32, srv954004393=97, srv1151037205=10, srv1618933697=31, srv185769862=40, srv689977312=79, srv86983396=89, srv1431037828=25, srv772403154=84, srv2088283381=56, srv1018458968=1, srv1432153063=26, srv935813467=94, srv297717738=63, srv1995569149=52, srv1177089415=11, srv987425846=99, srv1185043876=12, srv8684563=88, srv2069744653=55, srv347007415=67, srv854372046=87, srv1062181387=6, srv1052727655=5, srv615912247=75, srv110406127=8, srv1803779942=38, srv1905967899=44, srv1632620530=33, srv1366317408=23, srv306041533=64, srv697891374=80, srv91936946=91} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-17 09:53:01,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-17 09:53:01,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-17 09:53:01,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-17 09:53:01,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-17 09:53:01,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-17 09:53:01,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-17 09:53:01,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-17 09:53:01,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-17 09:53:01,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-17 09:53:01,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-17 09:53:01,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-17 09:53:01,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-17 09:53:01,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-17 09:53:01,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-17 09:53:01,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-17 09:53:01,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-17 09:53:01,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-17 09:53:01,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-17 09:53:01,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-17 09:53:01,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-17 09:53:01,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-17 09:53:01,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-17 09:53:01,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-17 09:53:01,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-17 09:53:01,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-17 09:53:01,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-17 09:53:01,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-17 09:53:01,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-17 09:53:01,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-17 09:53:01,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-17 09:53:01,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-17 09:53:01,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-17 09:53:01,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-17 09:53:01,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-17 09:53:01,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-17 09:53:01,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-17 09:53:01,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-17 09:53:01,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-17 09:53:01,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-17 09:53:01,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-17 09:53:01,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-17 09:53:01,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-17 09:53:01,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-17 09:53:01,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-17 09:53:01,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-17 09:53:01,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-17 09:53:01,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-17 09:53:01,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-17 09:53:01,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-17 09:53:01,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-17 09:53:01,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-17 09:53:01,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-17 09:53:01,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-17 09:53:01,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-17 09:53:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-17 09:53:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-17 09:53:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-17 09:53:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-17 09:53:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-17 09:53:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-17 09:53:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-17 09:53:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-17 09:53:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-17 09:53:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-17 09:53:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-17 09:53:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-17 09:53:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-17 09:53:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-17 09:53:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-17 09:53:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-17 09:53:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-17 09:53:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-17 09:53:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-17 09:53:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-17 09:53:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-17 09:53:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-17 09:53:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-17 09:53:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-17 09:53:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-17 09:53:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-17 09:53:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-17 09:53:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-17 09:53:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-17 09:53:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-17 09:53:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-17 09:53:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-17 09:53:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-17 09:53:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-17 09:53:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-17 09:53:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-17 09:53:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-17 09:53:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-17 09:53:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-17 09:53:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-17 09:53:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-17 09:53:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-17 09:53:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-17 09:53:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-17 09:53:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-17 09:53:01,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-17 09:53:01,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-17 09:53:02,251 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 09:53:02,251 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-17 09:53:02,251 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-17 09:53:02,654 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 765 ms to try 5000 different iterations. Found a solution that moves 4720 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17868536023198367. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041912146312016285, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05514596486668305, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5244444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041912146312016285, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17383333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24108333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 09:53:02,669 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv185769862:80 , srv2048131359:80 , srv772403154:80 , srv1185043876:81 , srv1262079392:81 , srv1177089415:82 , srv1223951671:82 , srv1046736216:83 , srv1052727655:83 , srv1062181387:83 , srv1195959051:83 , srv1216700836:83 , srv1728722955:83 , srv1743992913:83 , srv2029675636:83 , srv2133296283:83 , srv402330108:83 , srv91387693:83 , srv1447918517:84 , srv1657001475:84 , srv1803779942:84 , srv1880109519:84 , srv1964988541:84 , srv22586836:84 , srv245066002:84 , srv339249551:84 , srv528327842:84 , srv949362644:84 , srv1078158888:85 , srv1151037205:85 , srv1326989513:85 , srv1335954299:85 , srv1412421884:85 , srv1431037828:85 , srv1466361118:85 , srv1618933697:85 , srv1625769908:85 , srv1886711737:85 , srv208990820:85 , srv2133615522:85 , srv297717738:85 , srv367872298:85 , srv467101838:85 , srv745455391:85 , srv935813467:85 , srv987089988:85 , srv1129960133:86 , srv1310228683:86 , srv1317620640:86 , srv1879329479:86 , srv192262889:86 , srv1923985146:86 , srv1950507900:86 , srv502454678:86 , srv508607174:86 , srv702459095:86 , srv755603190:86 , srv854372046:86 , srv91936946:86 , srv1005373423:87 , srv1432153063:87 , srv1905967899:87 , srv280870525:87 , srv306041533:87 , srv689977312:87 , srv8684563:87 , srv924022310:87 , srv987425846:87 , srv1018458968:88 , srv1366317408:88 , srv1519022572:88 , srv1543904210:88 , srv1632620530:88 , srv347007415:88 , srv615912247:88 , srv791082659:88 , srv954004393:88 , srv1032248280:89 , srv1043092270:89 , srv1193092678:89 , srv1963311855:89 , srv1995569149:89 , srv2069744653:89 , srv341645461:89 , srv422237790:89 , srv697891374:89 , srv851034220:89 , srv943154066:89 , srv110406127:90 , srv1828591227:90 , srv628968785:90 , srv644993989:90 , srv933614630:91 , srv1948656322:92 , srv1962166509:92 , srv1691631250:93 , srv2088283381:93 , srv651845654:93 , srv86983396:103 , srv120557424:462 } 2023-02-17 09:53:02,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-17 09:53:02,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1262079392=18, srv528327842=74, srv1447918517=27, srv1129960133=9, srv1880109519=42, srv1964988541=51, srv1963311855=50, srv1962166509=49, srv502454678=72, srv1886711737=43, srv245066002=61, srv1195959051=14, srv1317620640=20, srv628968785=76, srv1657001475=34, srv1948656322=47, srv1005373423=0, srv1193092678=13, srv1691631250=35, srv702459095=81, srv755603190=83, srv1879329479=41, srv1043092270=3, srv508607174=73, srv943154066=95, srv1466361118=28, srv341645461=66, srv467101838=71, srv924022310=92, srv1078158888=7, srv367872298=68, srv1046736216=4, srv745455391=82, srv402330108=69, srv644993989=77, srv1032248280=2, srv91387693=90, srv2048131359=54, srv2133296283=58, srv791082659=85, srv1223951671=17, srv1519022572=29, srv1743992913=37, srv949362644=96, srv1412421884=24, srv120557424=15, srv1923985146=46, srv1326989513=21, srv192262889=45, srv280870525=62, srv22586836=60, srv1310228683=19, srv339249551=65, srv1543904210=30, srv208990820=57, srv933614630=93, srv1828591227=39, srv987089988=98, srv422237790=70, srv851034220=86, srv1728722955=36, srv1216700836=16, srv2133615522=59, srv1950507900=48, srv2029675636=53, srv651845654=78, srv1335954299=22, srv1625769908=32, srv954004393=97, srv1151037205=10, srv1618933697=31, srv185769862=40, srv689977312=79, srv86983396=89, srv1431037828=25, srv772403154=84, srv2088283381=56, srv1018458968=1, srv1432153063=26, srv935813467=94, srv297717738=63, srv1995569149=52, srv1177089415=11, srv987425846=99, srv1185043876=12, srv8684563=88, srv2069744653=55, srv347007415=67, srv854372046=87, srv1062181387=6, srv1052727655=5, srv615912247=75, srv110406127=8, srv1803779942=38, srv1905967899=44, srv1632620530=33, srv1366317408=23, srv306041533=64, srv697891374=80, srv91936946=91} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-17 09:53:02,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-17 09:53:02,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-17 09:53:02,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-17 09:53:02,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-17 09:53:02,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-17 09:53:02,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-17 09:53:02,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-17 09:53:02,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-17 09:53:02,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-17 09:53:02,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-17 09:53:02,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-17 09:53:02,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-17 09:53:02,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-17 09:53:02,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-17 09:53:02,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-17 09:53:02,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-17 09:53:02,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-17 09:53:02,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-17 09:53:02,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-17 09:53:02,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-17 09:53:02,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-17 09:53:02,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-17 09:53:02,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-17 09:53:02,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-17 09:53:02,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-17 09:53:02,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-17 09:53:02,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-17 09:53:02,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-17 09:53:02,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-17 09:53:02,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-17 09:53:02,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-17 09:53:02,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-17 09:53:02,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-17 09:53:02,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-17 09:53:02,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-17 09:53:02,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-17 09:53:02,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-17 09:53:02,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-17 09:53:02,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-17 09:53:02,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-17 09:53:02,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-17 09:53:02,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-17 09:53:02,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-17 09:53:02,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-17 09:53:02,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-17 09:53:02,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-17 09:53:02,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-17 09:53:02,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-17 09:53:02,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-17 09:53:02,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-17 09:53:02,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-17 09:53:02,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-17 09:53:02,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-17 09:53:02,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-17 09:53:02,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-17 09:53:02,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-17 09:53:02,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-17 09:53:02,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-17 09:53:02,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-17 09:53:02,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-17 09:53:02,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-17 09:53:02,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-17 09:53:02,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-17 09:53:02,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-17 09:53:02,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-17 09:53:02,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-17 09:53:02,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-17 09:53:02,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-17 09:53:02,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-17 09:53:02,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-17 09:53:02,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-17 09:53:02,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-17 09:53:02,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-17 09:53:02,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-17 09:53:02,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-17 09:53:02,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-17 09:53:02,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-17 09:53:02,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-17 09:53:02,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-17 09:53:02,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-17 09:53:02,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-17 09:53:02,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-17 09:53:02,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-17 09:53:02,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-17 09:53:02,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-17 09:53:02,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-17 09:53:02,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-17 09:53:02,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-17 09:53:02,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-17 09:53:02,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-17 09:53:02,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-17 09:53:02,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-17 09:53:02,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-17 09:53:02,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-17 09:53:02,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-17 09:53:02,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-17 09:53:02,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-17 09:53:02,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-17 09:53:02,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-17 09:53:02,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-17 09:53:02,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-17 09:53:02,830 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041912146312016285, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05514596486668305, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041912146312016285, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17383333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24108333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 09:53:02,830 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-17 09:53:02,830 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17865231008558874, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041912146312016285, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05514596486668305, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041912146312016285, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17383333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24108333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-17 09:53:03,028 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 295 ms to try 5000 different iterations. Found a solution that moves 2641 regions; Going from a computed imbalance of 0.17865231008558874 to a new imbalance of 0.01420020582606209. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02283007054768165); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03276279404142256, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29344444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02283007054768165); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 09:53:03,036 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv86983396:70 , srv1962166509:78 , srv1828591227:80 , srv628968785:80 , srv1691631250:81 , srv1948656322:81 , srv2069744653:81 , srv2088283381:81 , srv644993989:81 , srv651845654:81 , srv697891374:81 , srv933614630:81 , srv1043092270:82 , srv110406127:82 , srv1032248280:83 , srv1519022572:83 , srv347007415:83 , srv422237790:83 , srv615912247:83 , srv689977312:83 , srv791082659:83 , srv943154066:83 , srv1005373423:84 , srv1193092678:84 , srv1543904210:84 , srv1963311855:84 , srv306041533:84 , srv341645461:84 , srv702459095:84 , srv987425846:84 , srv1326989513:85 , srv1618933697:85 , srv1632620530:85 , srv192262889:85 , srv1923985146:85 , srv1950507900:85 , srv1995569149:85 , srv755603190:85 , srv91936946:85 , srv935813467:85 , srv954004393:85 , srv1018458968:86 , srv1078158888:86 , srv1310228683:86 , srv1432153063:86 , srv1886711737:86 , srv2133615522:86 , srv280870525:86 , srv528327842:86 , srv851034220:86 , srv1317620640:87 , srv1335954299:87 , srv1366317408:87 , srv1412421884:87 , srv1431037828:87 , srv1728722955:87 , srv1803779942:87 , srv1905967899:87 , srv208990820:87 , srv22586836:87 , srv245066002:87 , srv402330108:87 , srv467101838:87 , srv502454678:87 , srv508607174:87 , srv854372046:87 , srv924022310:87 , srv987089988:87 , srv1052727655:88 , srv1129960133:88 , srv1151037205:88 , srv1216700836:88 , srv1625769908:88 , srv1657001475:88 , srv1743992913:88 , srv1879329479:88 , srv1880109519:88 , srv1964988541:88 , srv2029675636:88 , srv2133296283:88 , srv297717738:88 , srv339249551:88 , srv367872298:88 , srv745455391:88 , srv8684563:88 , srv1046736216:89 , srv1062181387:89 , srv1177089415:89 , srv1223951671:89 , srv1447918517:89 , srv1466361118:89 , srv91387693:89 , srv949362644:89 , srv1195959051:90 , srv185769862:90 , srv1185043876:91 , srv1262079392:91 , srv2048131359:92 , srv772403154:92 , srv120557424:515 } 2023-02-17 09:53:03,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-17 09:53:03,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1262079392=18, srv528327842=74, srv1447918517=27, srv1129960133=9, srv1880109519=42, srv1964988541=51, srv1963311855=50, srv1962166509=49, srv502454678=72, srv1886711737=43, srv245066002=61, srv1195959051=14, srv1317620640=20, srv628968785=76, srv1657001475=34, srv1948656322=47, srv1005373423=0, srv1193092678=13, srv1691631250=35, srv702459095=81, srv755603190=83, srv1879329479=41, srv1043092270=3, srv508607174=73, srv943154066=95, srv1466361118=28, srv341645461=66, srv467101838=71, srv924022310=92, srv1078158888=7, srv367872298=68, srv1046736216=4, srv745455391=82, srv402330108=69, srv644993989=77, srv1032248280=2, srv91387693=90, srv2048131359=54, srv2133296283=58, srv791082659=85, srv1223951671=17, srv1519022572=29, srv1743992913=37, srv949362644=96, srv1412421884=24, srv120557424=15, srv1923985146=46, srv1326989513=21, srv192262889=45, srv280870525=62, srv22586836=60, srv1310228683=19, srv339249551=65, srv1543904210=30, srv208990820=57, srv933614630=93, srv1828591227=39, srv987089988=98, srv422237790=70, srv851034220=86, srv1728722955=36, srv1216700836=16, srv2133615522=59, srv1950507900=48, srv2029675636=53, srv651845654=78, srv1335954299=22, srv1625769908=32, srv954004393=97, srv1151037205=10, srv1618933697=31, srv185769862=40, srv689977312=79, srv86983396=89, srv1431037828=25, srv772403154=84, srv2088283381=56, srv1018458968=1, srv1432153063=26, srv935813467=94, srv297717738=63, srv1995569149=52, srv1177089415=11, srv987425846=99, srv1185043876=12, srv8684563=88, srv2069744653=55, srv347007415=67, srv854372046=87, srv1062181387=6, srv1052727655=5, srv615912247=75, srv110406127=8, srv1803779942=38, srv1905967899=44, srv1632620530=33, srv1366317408=23, srv306041533=64, srv697891374=80, srv91936946=91} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-17 09:53:03,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-17 09:53:03,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-17 09:53:03,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-17 09:53:03,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-17 09:53:03,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-17 09:53:03,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-17 09:53:03,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-17 09:53:03,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-17 09:53:03,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-17 09:53:03,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-17 09:53:03,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-17 09:53:03,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-17 09:53:03,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-17 09:53:03,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-17 09:53:03,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-17 09:53:03,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-17 09:53:03,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-17 09:53:03,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-17 09:53:03,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-17 09:53:03,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-17 09:53:03,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-17 09:53:03,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-17 09:53:03,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-17 09:53:03,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-17 09:53:03,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-17 09:53:03,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-17 09:53:03,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-17 09:53:03,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-17 09:53:03,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-17 09:53:03,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-17 09:53:03,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-17 09:53:03,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-17 09:53:03,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-17 09:53:03,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-17 09:53:03,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-17 09:53:03,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-17 09:53:03,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-17 09:53:03,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-17 09:53:03,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-17 09:53:03,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-17 09:53:03,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-17 09:53:03,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-17 09:53:03,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-17 09:53:03,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-17 09:53:03,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-17 09:53:03,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-17 09:53:03,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-17 09:53:03,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-17 09:53:03,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-17 09:53:03,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-17 09:53:03,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-17 09:53:03,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-17 09:53:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-17 09:53:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-17 09:53:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-17 09:53:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-17 09:53:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-17 09:53:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-17 09:53:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-17 09:53:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-17 09:53:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-17 09:53:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-17 09:53:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-17 09:53:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-17 09:53:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-17 09:53:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-17 09:53:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-17 09:53:03,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-17 09:53:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-17 09:53:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-17 09:53:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-17 09:53:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-17 09:53:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-17 09:53:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-17 09:53:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-17 09:53:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-17 09:53:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-17 09:53:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-17 09:53:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-17 09:53:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-17 09:53:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-17 09:53:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-17 09:53:03,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-17 09:53:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-17 09:53:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-17 09:53:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-17 09:53:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-17 09:53:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-17 09:53:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-17 09:53:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-17 09:53:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-17 09:53:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-17 09:53:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-17 09:53:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-17 09:53:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-17 09:53:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-17 09:53:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-17 09:53:03,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-17 09:53:03,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-17 09:53:03,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-17 09:53:03,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-17 09:53:03,142 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02283007054768165); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03276279404142256, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02283007054768165); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 09:53:03,143 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-17 09:53:03,143 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014181713148810175, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02283007054768165); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03276279404142256, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02283007054768165); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-17 09:53:03,316 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 243 ms to try 5000 different iterations. Found a solution that moves 889 regions; Going from a computed imbalance of 0.014181713148810175 to a new imbalance of 6.213657624110733E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008447220289230645); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009284141650970551); MoveCostFunction : (multiplier=7.0, imbalance=0.09877777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008447220289230645); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 09:53:03,318 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1018458968:83 , srv1366317408:83 , srv1466361118:83 , srv1691631250:83 , srv2088283381:83 , srv651845654:83 , srv851034220:83 , srv8684563:83 , srv86983396:83 , srv1129960133:84 , srv1151037205:84 , srv1195959051:84 , srv1432153063:84 , srv1632620530:84 , srv1657001475:84 , srv1879329479:84 , srv1880109519:84 , srv1905967899:84 , srv1948656322:84 , srv1995569149:84 , srv297717738:84 , srv341645461:84 , srv508607174:84 , srv745455391:84 , srv854372046:84 , srv924022310:84 , srv949362644:84 , srv1032248280:85 , srv1043092270:85 , srv1046736216:85 , srv1062181387:85 , srv110406127:85 , srv1185043876:85 , srv1193092678:85 , srv1262079392:85 , srv1310228683:85 , srv1317620640:85 , srv1335954299:85 , srv1431037828:85 , srv1447918517:85 , srv1543904210:85 , srv1625769908:85 , srv1803779942:85 , srv1950507900:85 , srv1963311855:85 , srv1964988541:85 , srv2133615522:85 , srv280870525:85 , srv306041533:85 , srv367872298:85 , srv422237790:85 , srv467101838:85 , srv502454678:85 , srv755603190:85 , srv954004393:85 , srv1052727655:86 , srv1078158888:86 , srv1177089415:86 , srv1216700836:86 , srv1223951671:86 , srv1412421884:86 , srv1519022572:86 , srv1886711737:86 , srv192262889:86 , srv2029675636:86 , srv2048131359:86 , srv208990820:86 , srv22586836:86 , srv339249551:86 , srv347007415:86 , srv528327842:86 , srv615912247:86 , srv628968785:86 , srv772403154:86 , srv791082659:86 , srv91387693:86 , srv91936946:86 , srv933614630:86 , srv943154066:86 , srv987089988:86 , srv987425846:86 , srv1005373423:87 , srv1326989513:87 , srv1618933697:87 , srv1743992913:87 , srv1828591227:87 , srv185769862:87 , srv1923985146:87 , srv2069744653:87 , srv2133296283:87 , srv245066002:87 , srv402330108:87 , srv644993989:87 , srv689977312:87 , srv697891374:87 , srv702459095:87 , srv1728722955:88 , srv1962166509:88 , srv935813467:88 , srv120557424:556 } 2023-02-17 09:53:03,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-17 09:53:03,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1262079392=18, srv528327842=74, srv1447918517=27, srv1129960133=9, srv1880109519=42, srv1964988541=51, srv1963311855=50, srv1962166509=49, srv502454678=72, srv1886711737=43, srv245066002=61, srv1195959051=14, srv1317620640=20, srv628968785=76, srv1657001475=34, srv1948656322=47, srv1005373423=0, srv1193092678=13, srv1691631250=35, srv702459095=81, srv755603190=83, srv1879329479=41, srv1043092270=3, srv508607174=73, srv943154066=95, srv1466361118=28, srv341645461=66, srv467101838=71, srv924022310=92, srv1078158888=7, srv367872298=68, srv1046736216=4, srv745455391=82, srv402330108=69, srv644993989=77, srv1032248280=2, srv91387693=90, srv2048131359=54, srv2133296283=58, srv791082659=85, srv1223951671=17, srv1519022572=29, srv1743992913=37, srv949362644=96, srv1412421884=24, srv120557424=15, srv1923985146=46, srv1326989513=21, srv192262889=45, srv280870525=62, srv22586836=60, srv1310228683=19, srv339249551=65, srv1543904210=30, srv208990820=57, srv933614630=93, srv1828591227=39, srv987089988=98, srv422237790=70, srv851034220=86, srv1728722955=36, srv1216700836=16, srv2133615522=59, srv1950507900=48, srv2029675636=53, srv651845654=78, srv1335954299=22, srv1625769908=32, srv954004393=97, srv1151037205=10, srv1618933697=31, srv185769862=40, srv689977312=79, srv86983396=89, srv1431037828=25, srv772403154=84, srv2088283381=56, srv1018458968=1, srv1432153063=26, srv935813467=94, srv297717738=63, srv1995569149=52, srv1177089415=11, srv987425846=99, srv1185043876=12, srv8684563=88, srv2069744653=55, srv347007415=67, srv854372046=87, srv1062181387=6, srv1052727655=5, srv615912247=75, srv110406127=8, srv1803779942=38, srv1905967899=44, srv1632620530=33, srv1366317408=23, srv306041533=64, srv697891374=80, srv91936946=91} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-17 09:53:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-17 09:53:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-17 09:53:03,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-17 09:53:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-17 09:53:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-17 09:53:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-17 09:53:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-17 09:53:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-17 09:53:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-17 09:53:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-17 09:53:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-17 09:53:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-17 09:53:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-17 09:53:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-17 09:53:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-17 09:53:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-17 09:53:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-17 09:53:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-17 09:53:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-17 09:53:03,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-17 09:53:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-17 09:53:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-17 09:53:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-17 09:53:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-17 09:53:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-17 09:53:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-17 09:53:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-17 09:53:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-17 09:53:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-17 09:53:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-17 09:53:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-17 09:53:03,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-17 09:53:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-17 09:53:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-17 09:53:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-17 09:53:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-17 09:53:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-17 09:53:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-17 09:53:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-17 09:53:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-17 09:53:03,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-17 09:53:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-17 09:53:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-17 09:53:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-17 09:53:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-17 09:53:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-17 09:53:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-17 09:53:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-17 09:53:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-17 09:53:03,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-17 09:53:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-17 09:53:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-17 09:53:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-17 09:53:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-17 09:53:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-17 09:53:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-17 09:53:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-17 09:53:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-17 09:53:03,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-17 09:53:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-17 09:53:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-17 09:53:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-17 09:53:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-17 09:53:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-17 09:53:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-17 09:53:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-17 09:53:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-17 09:53:03,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-17 09:53:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-17 09:53:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-17 09:53:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-17 09:53:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-17 09:53:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-17 09:53:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-17 09:53:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-17 09:53:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-17 09:53:03,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-17 09:53:03,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-17 09:53:03,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-17 09:53:03,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-17 09:53:03,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-17 09:53:03,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-17 09:53:03,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-17 09:53:03,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-17 09:53:03,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-17 09:53:03,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-17 09:53:03,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-17 09:53:03,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-17 09:53:03,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-17 09:53:03,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-17 09:53:03,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-17 09:53:03,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-17 09:53:03,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-17 09:53:03,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-17 09:53:03,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-17 09:53:03,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-17 09:53:03,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-17 09:53:03,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-17 09:53:03,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-17 09:53:03,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-17 09:53:03,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-17 09:53:03,419 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008447220289230645); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009284141650970551); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008447220289230645); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 09:53:03,419 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-17 09:53:03,419 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.151408513633816E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008447220289230645); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009284141650970551); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008447220289230645); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-17 09:53:03,636 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 288 ms to try 5000 different iterations. Found a solution that moves 240 regions; Going from a computed imbalance of 6.151408513633816E-4 to a new imbalance of 3.48020824984594E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.02666666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 09:53:03,637 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1366317408:83 , srv1447918517:83 , srv1005373423:84 , srv1052727655:84 , srv1129960133:84 , srv1177089415:84 , srv1193092678:84 , srv1412421884:84 , srv1543904210:84 , srv1618933697:84 , srv1691631250:84 , srv1743992913:84 , srv1828591227:84 , srv1879329479:84 , srv1963311855:84 , srv1995569149:84 , srv2048131359:84 , srv2133296283:84 , srv245066002:84 , srv280870525:84 , srv339249551:84 , srv347007415:84 , srv367872298:84 , srv502454678:84 , srv772403154:84 , srv851034220:84 , srv8684563:84 , srv91387693:84 , srv91936946:84 , srv924022310:84 , srv933614630:84 , srv935813467:84 , srv943154066:84 , srv954004393:84 , srv987425846:84 , srv1018458968:85 , srv1032248280:85 , srv1046736216:85 , srv1062181387:85 , srv1078158888:85 , srv1151037205:85 , srv1185043876:85 , srv1195959051:85 , srv1216700836:85 , srv1223951671:85 , srv1262079392:85 , srv1310228683:85 , srv1317620640:85 , srv1326989513:85 , srv1335954299:85 , srv1431037828:85 , srv1432153063:85 , srv1519022572:85 , srv1625769908:85 , srv1632620530:85 , srv1728722955:85 , srv185769862:85 , srv1886711737:85 , srv1905967899:85 , srv192262889:85 , srv1923985146:85 , srv1948656322:85 , srv1962166509:85 , srv1964988541:85 , srv2029675636:85 , srv2069744653:85 , srv2088283381:85 , srv208990820:85 , srv2133615522:85 , srv22586836:85 , srv297717738:85 , srv341645461:85 , srv402330108:85 , srv422237790:85 , srv467101838:85 , srv508607174:85 , srv615912247:85 , srv644993989:85 , srv651845654:85 , srv689977312:85 , srv697891374:85 , srv702459095:85 , srv745455391:85 , srv791082659:85 , srv854372046:85 , srv949362644:85 , srv987089988:85 , srv1043092270:86 , srv110406127:86 , srv1466361118:86 , srv1657001475:86 , srv1803779942:86 , srv1880109519:86 , srv1950507900:86 , srv306041533:86 , srv628968785:86 , srv755603190:86 , srv86983396:86 , srv528327842:87 , srv120557424:609 } 2023-02-17 09:53:03,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-17 09:53:03,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1262079392=18, srv528327842=74, srv1447918517=27, srv1129960133=9, srv1880109519=42, srv1964988541=51, srv1963311855=50, srv1962166509=49, srv502454678=72, srv1886711737=43, srv245066002=61, srv1195959051=14, srv1317620640=20, srv628968785=76, srv1657001475=34, srv1948656322=47, srv1005373423=0, srv1193092678=13, srv1691631250=35, srv702459095=81, srv755603190=83, srv1879329479=41, srv1043092270=3, srv508607174=73, srv943154066=95, srv1466361118=28, srv341645461=66, srv467101838=71, srv924022310=92, srv1078158888=7, srv367872298=68, srv1046736216=4, srv745455391=82, srv402330108=69, srv644993989=77, srv1032248280=2, srv91387693=90, srv2048131359=54, srv2133296283=58, srv791082659=85, srv1223951671=17, srv1519022572=29, srv1743992913=37, srv949362644=96, srv1412421884=24, srv120557424=15, srv1923985146=46, srv1326989513=21, srv192262889=45, srv280870525=62, srv22586836=60, srv1310228683=19, srv339249551=65, srv1543904210=30, srv208990820=57, srv933614630=93, srv1828591227=39, srv987089988=98, srv422237790=70, srv851034220=86, srv1728722955=36, srv1216700836=16, srv2133615522=59, srv1950507900=48, srv2029675636=53, srv651845654=78, srv1335954299=22, srv1625769908=32, srv954004393=97, srv1151037205=10, srv1618933697=31, srv185769862=40, srv689977312=79, srv86983396=89, srv1431037828=25, srv772403154=84, srv2088283381=56, srv1018458968=1, srv1432153063=26, srv935813467=94, srv297717738=63, srv1995569149=52, srv1177089415=11, srv987425846=99, srv1185043876=12, srv8684563=88, srv2069744653=55, srv347007415=67, srv854372046=87, srv1062181387=6, srv1052727655=5, srv615912247=75, srv110406127=8, srv1803779942=38, srv1905967899=44, srv1632620530=33, srv1366317408=23, srv306041533=64, srv697891374=80, srv91936946=91} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-17 09:53:03,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-17 09:53:03,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-17 09:53:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-17 09:53:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-17 09:53:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-17 09:53:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-17 09:53:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-17 09:53:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-17 09:53:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-17 09:53:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-17 09:53:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-17 09:53:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-17 09:53:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-17 09:53:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-17 09:53:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-17 09:53:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-17 09:53:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-17 09:53:03,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-17 09:53:03,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-17 09:53:03,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-17 09:53:03,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-17 09:53:03,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-17 09:53:03,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-17 09:53:03,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-17 09:53:03,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-17 09:53:03,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-17 09:53:03,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-17 09:53:03,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-17 09:53:03,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-17 09:53:03,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-17 09:53:03,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-17 09:53:03,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-17 09:53:03,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-17 09:53:03,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-17 09:53:03,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-17 09:53:03,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-17 09:53:03,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-17 09:53:03,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-17 09:53:03,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-17 09:53:03,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-17 09:53:03,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-17 09:53:03,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-17 09:53:03,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-17 09:53:03,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-17 09:53:03,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-17 09:53:03,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-17 09:53:03,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-17 09:53:03,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-17 09:53:03,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-17 09:53:03,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-17 09:53:03,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-17 09:53:03,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-17 09:53:03,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-17 09:53:03,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-17 09:53:03,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-17 09:53:03,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-17 09:53:03,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-17 09:53:03,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-17 09:53:03,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-17 09:53:03,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-17 09:53:03,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-17 09:53:03,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-17 09:53:03,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-17 09:53:03,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-17 09:53:03,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-17 09:53:03,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-17 09:53:03,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-17 09:53:03,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-17 09:53:03,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-17 09:53:03,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-17 09:53:03,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-17 09:53:03,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-17 09:53:03,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-17 09:53:03,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-17 09:53:03,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-17 09:53:03,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-17 09:53:03,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-17 09:53:03,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-17 09:53:03,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-17 09:53:03,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-17 09:53:03,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-17 09:53:03,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-17 09:53:03,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-17 09:53:03,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-17 09:53:03,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-17 09:53:03,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-17 09:53:03,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-17 09:53:03,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-17 09:53:03,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-17 09:53:03,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-17 09:53:03,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-17 09:53:03,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-17 09:53:03,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-17 09:53:03,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-17 09:53:03,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-17 09:53:03,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-17 09:53:03,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-17 09:53:03,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-17 09:53:03,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-17 09:53:03,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-17 09:53:03,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-17 09:53:03,744 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 09:53:03,744 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-17 09:53:03,745 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.312156658007246E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-17 09:53:03,887 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 216 ms to try 5000 different iterations. Found a solution that moves 22 regions; Going from a computed imbalance of 3.312156658007246E-5 to a new imbalance of 7.541804792831362E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0024444444444444444); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 09:53:03,888 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv110406127:83 , srv1317620640:83 , srv1466361118:83 , srv1625769908:83 , srv1728722955:83 , srv1905967899:83 , srv1923985146:83 , srv1962166509:83 , srv208990820:83 , srv528327842:83 , srv644993989:83 , srv987089988:83 , srv1005373423:84 , srv1018458968:84 , srv1032248280:84 , srv1043092270:84 , srv1046736216:84 , srv1062181387:84 , srv1078158888:84 , srv1129960133:84 , srv1151037205:84 , srv1185043876:84 , srv1193092678:84 , srv1195959051:84 , srv1216700836:84 , srv1223951671:84 , srv1262079392:84 , srv1310228683:84 , srv1326989513:84 , srv1335954299:84 , srv1412421884:84 , srv1431037828:84 , srv1432153063:84 , srv1519022572:84 , srv1632620530:84 , srv1657001475:84 , srv1743992913:84 , srv1803779942:84 , srv185769862:84 , srv1879329479:84 , srv1880109519:84 , srv1886711737:84 , srv192262889:84 , srv1948656322:84 , srv1950507900:84 , srv1963311855:84 , srv1964988541:84 , srv1995569149:84 , srv2029675636:84 , srv2048131359:84 , srv2069744653:84 , srv2088283381:84 , srv2133296283:84 , srv22586836:84 , srv245066002:84 , srv280870525:84 , srv297717738:84 , srv306041533:84 , srv339249551:84 , srv341645461:84 , srv367872298:84 , srv402330108:84 , srv422237790:84 , srv467101838:84 , srv502454678:84 , srv508607174:84 , srv615912247:84 , srv628968785:84 , srv651845654:84 , srv689977312:84 , srv697891374:84 , srv702459095:84 , srv745455391:84 , srv755603190:84 , srv772403154:84 , srv791082659:84 , srv851034220:84 , srv854372046:84 , srv8684563:84 , srv86983396:84 , srv91387693:84 , srv924022310:84 , srv933614630:84 , srv935813467:84 , srv943154066:84 , srv949362644:84 , srv954004393:84 , srv1052727655:85 , srv1177089415:85 , srv1366317408:85 , srv1447918517:85 , srv1543904210:85 , srv1618933697:85 , srv1691631250:85 , srv1828591227:85 , srv2133615522:85 , srv347007415:85 , srv91936946:85 , srv987425846:85 , srv120557424:684 } 2023-02-17 09:53:03,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-17 09:53:03,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1262079392=18, srv528327842=74, srv1447918517=27, srv1129960133=9, srv1880109519=42, srv1964988541=51, srv1963311855=50, srv1962166509=49, srv502454678=72, srv1886711737=43, srv245066002=61, srv1195959051=14, srv1317620640=20, srv628968785=76, srv1657001475=34, srv1948656322=47, srv1005373423=0, srv1193092678=13, srv1691631250=35, srv702459095=81, srv755603190=83, srv1879329479=41, srv1043092270=3, srv508607174=73, srv943154066=95, srv1466361118=28, srv341645461=66, srv467101838=71, srv924022310=92, srv1078158888=7, srv367872298=68, srv1046736216=4, srv745455391=82, srv402330108=69, srv644993989=77, srv1032248280=2, srv91387693=90, srv2048131359=54, srv2133296283=58, srv791082659=85, srv1223951671=17, srv1519022572=29, srv1743992913=37, srv949362644=96, srv1412421884=24, srv120557424=15, srv1923985146=46, srv1326989513=21, srv192262889=45, srv280870525=62, srv22586836=60, srv1310228683=19, srv339249551=65, srv1543904210=30, srv208990820=57, srv933614630=93, srv1828591227=39, srv987089988=98, srv422237790=70, srv851034220=86, srv1728722955=36, srv1216700836=16, srv2133615522=59, srv1950507900=48, srv2029675636=53, srv651845654=78, srv1335954299=22, srv1625769908=32, srv954004393=97, srv1151037205=10, srv1618933697=31, srv185769862=40, srv689977312=79, srv86983396=89, srv1431037828=25, srv772403154=84, srv2088283381=56, srv1018458968=1, srv1432153063=26, srv935813467=94, srv297717738=63, srv1995569149=52, srv1177089415=11, srv987425846=99, srv1185043876=12, srv8684563=88, srv2069744653=55, srv347007415=67, srv854372046=87, srv1062181387=6, srv1052727655=5, srv615912247=75, srv110406127=8, srv1803779942=38, srv1905967899=44, srv1632620530=33, srv1366317408=23, srv306041533=64, srv697891374=80, srv91936946=91} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-17 09:53:03,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-17 09:53:03,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-17 09:53:03,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-17 09:53:03,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-17 09:53:03,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-17 09:53:03,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-17 09:53:03,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-17 09:53:03,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-17 09:53:03,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-17 09:53:03,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-17 09:53:03,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-17 09:53:03,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-17 09:53:03,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-17 09:53:03,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-17 09:53:03,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-17 09:53:03,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-17 09:53:03,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-17 09:53:03,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-17 09:53:03,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-17 09:53:03,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-17 09:53:03,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-17 09:53:03,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-17 09:53:03,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-17 09:53:03,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-17 09:53:03,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-17 09:53:03,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-17 09:53:03,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-17 09:53:03,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-17 09:53:03,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-17 09:53:03,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-17 09:53:03,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-17 09:53:03,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-17 09:53:03,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-17 09:53:03,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-17 09:53:03,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-17 09:53:03,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-17 09:53:03,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-17 09:53:03,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-17 09:53:03,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-17 09:53:03,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-17 09:53:03,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-17 09:53:03,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-17 09:53:03,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-17 09:53:03,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-17 09:53:03,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-17 09:53:03,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-17 09:53:03,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-17 09:53:03,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-17 09:53:03,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-17 09:53:03,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-17 09:53:03,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-17 09:53:03,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-17 09:53:03,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-17 09:53:03,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-17 09:53:03,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-17 09:53:03,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-17 09:53:03,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-17 09:53:03,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-17 09:53:03,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-17 09:53:03,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-17 09:53:03,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-17 09:53:03,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-17 09:53:03,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-17 09:53:03,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-17 09:53:03,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-17 09:53:03,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-17 09:53:03,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-17 09:53:03,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-17 09:53:03,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-17 09:53:03,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-17 09:53:03,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-17 09:53:03,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-17 09:53:03,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-17 09:53:03,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-17 09:53:03,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-17 09:53:03,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-17 09:53:03,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-17 09:53:03,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-17 09:53:03,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-17 09:53:03,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-17 09:53:03,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-17 09:53:03,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-17 09:53:03,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-17 09:53:03,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-17 09:53:03,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-17 09:53:03,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-17 09:53:03,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-17 09:53:03,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-17 09:53:03,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-17 09:53:03,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-17 09:53:03,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-17 09:53:03,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-17 09:53:03,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-17 09:53:03,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-17 09:53:03,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-17 09:53:03,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-17 09:53:03,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-17 09:53:03,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-17 09:53:03,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-17 09:53:03,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-17 09:53:03,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-17 09:53:03,990 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 09:53:03,990 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv110406127:83 , srv1317620640:83 , srv1466361118:83 , srv1625769908:83 , srv1728722955:83 , srv1905967899:83 , srv1923985146:83 , srv1962166509:83 , srv208990820:83 , srv528327842:83 , srv644993989:83 , srv987089988:83 , srv1005373423:84 , srv1018458968:84 , srv1032248280:84 , srv1043092270:84 , srv1046736216:84 , srv1062181387:84 , srv1078158888:84 , srv1129960133:84 , srv1151037205:84 , srv1185043876:84 , srv1193092678:84 , srv1195959051:84 , srv1216700836:84 , srv1223951671:84 , srv1262079392:84 , srv1310228683:84 , srv1326989513:84 , srv1335954299:84 , srv1412421884:84 , srv1431037828:84 , srv1432153063:84 , srv1519022572:84 , srv1632620530:84 , srv1657001475:84 , srv1743992913:84 , srv1803779942:84 , srv185769862:84 , srv1879329479:84 , srv1880109519:84 , srv1886711737:84 , srv192262889:84 , srv1948656322:84 , srv1950507900:84 , srv1963311855:84 , srv1964988541:84 , srv1995569149:84 , srv2029675636:84 , srv2048131359:84 , srv2069744653:84 , srv2088283381:84 , srv2133296283:84 , srv22586836:84 , srv245066002:84 , srv280870525:84 , srv297717738:84 , srv306041533:84 , srv339249551:84 , srv341645461:84 , srv367872298:84 , srv402330108:84 , srv422237790:84 , srv467101838:84 , srv502454678:84 , srv508607174:84 , srv615912247:84 , srv628968785:84 , srv651845654:84 , srv689977312:84 , srv697891374:84 , srv702459095:84 , srv745455391:84 , srv755603190:84 , srv772403154:84 , srv791082659:84 , srv851034220:84 , srv854372046:84 , srv8684563:84 , srv86983396:84 , srv91387693:84 , srv924022310:84 , srv933614630:84 , srv935813467:84 , srv943154066:84 , srv949362644:84 , srv954004393:84 , srv1052727655:85 , srv1177089415:85 , srv1366317408:85 , srv1447918517:85 , srv1543904210:85 , srv1618933697:85 , srv1691631250:85 , srv1828591227:85 , srv2133615522:85 , srv347007415:85 , srv91936946:85 , srv987425846:85 , srv120557424:684 } 2023-02-17 09:53:04,010 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=397 (was 379) - SystemLoadAverage LEAK? -, ProcessCount=174 (was 174), AvailableMemoryMB=3187 (was 3808) 2023-02-17 09:53:04,016 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=397, ProcessCount=174, AvailableMemoryMB=3186 2023-02-17 09:53:04,017 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-17 09:53:04,017 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-02-17 09:53:04,026 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1093174527:84 , srv1118631597:84 , srv1192731177:84 , srv1232137032:84 , srv1234533165:84 , srv1259957600:84 , srv1341502263:84 , srv1478243588:84 , srv148413063:84 , srv1489452889:84 , srv1574888719:84 , srv1607296757:84 , srv1678988329:84 , srv1766009553:84 , srv1787914209:84 , srv1861984539:84 , srv1957275895:84 , srv195851430:84 , srv1971247403:84 , srv2093812785:84 , srv2134640543:84 , srv223953116:84 , srv517355410:84 , srv534218916:84 , srv58975246:84 , srv714542438:84 , srv740368190:84 , srv747080202:84 , srv997832001:84 , srv2125357509:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-02-17 09:53:04,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-17 09:53:04,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1478243588=7, srv195851430=17, srv2125357509=20, srv223953116=22, srv1766009553=13, srv58975246=25, srv148413063=8, srv534218916=24, srv1259957600=5, srv1678988329=12, srv1607296757=11, srv747080202=28, srv2134640543=21, srv1787914209=14, srv1093174527=0, srv1574888719=10, srv517355410=23, srv1341502263=6, srv2093812785=19, srv1192731177=2, srv997832001=29, srv1234533165=4, srv1957275895=16, srv1971247403=18, srv1861984539=15, srv1118631597=1, srv1232137032=3, srv1489452889=9, srv740368190=27, srv714542438=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-17 09:53:04,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-17 09:53:04,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-17 09:53:04,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-17 09:53:04,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-17 09:53:04,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-17 09:53:04,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-17 09:53:04,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-17 09:53:04,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-17 09:53:04,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-17 09:53:04,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-17 09:53:04,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-17 09:53:04,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-17 09:53:04,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-17 09:53:04,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-17 09:53:04,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-17 09:53:04,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-17 09:53:04,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-17 09:53:04,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-17 09:53:04,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-17 09:53:04,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-17 09:53:04,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-17 09:53:04,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-17 09:53:04,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-17 09:53:04,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-17 09:53:04,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-17 09:53:04,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-17 09:53:04,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-17 09:53:04,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-17 09:53:04,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-17 09:53:04,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-17 09:53:04,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-17 09:53:04,051 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1279208874706572, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 09:53:04,051 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912596117803698, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1279208874706572, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-02-17 09:55:04,041 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9556542 different iterations. Found a solution that moves 1837 regions; Going from a computed imbalance of 0.9912596117803698 to a new imbalance of 4.2876496417733574E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6803703703703704, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 09:55:04,046 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1093174527:90 , srv1118631597:90 , srv1192731177:90 , srv1232137032:90 , srv1234533165:90 , srv1259957600:90 , srv1341502263:90 , srv1478243588:90 , srv148413063:90 , srv1489452889:90 , srv1574888719:90 , srv1607296757:90 , srv1678988329:90 , srv1766009553:90 , srv1787914209:90 , srv1861984539:90 , srv1957275895:90 , srv195851430:90 , srv1971247403:90 , srv2093812785:90 , srv2125357509:90 , srv2134640543:90 , srv223953116:90 , srv517355410:90 , srv534218916:90 , srv58975246:90 , srv714542438:90 , srv740368190:90 , srv747080202:90 , srv997832001:90 } 2023-02-17 09:55:04,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-17 09:55:04,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1478243588=7, srv195851430=17, srv2125357509=20, srv223953116=22, srv1766009553=13, srv58975246=25, srv148413063=8, srv534218916=24, srv1259957600=5, srv1678988329=12, srv1607296757=11, srv747080202=28, srv2134640543=21, srv1787914209=14, srv1093174527=0, srv1574888719=10, srv517355410=23, srv1341502263=6, srv2093812785=19, srv1192731177=2, srv997832001=29, srv1234533165=4, srv1957275895=16, srv1971247403=18, srv1861984539=15, srv1118631597=1, srv1232137032=3, srv1489452889=9, srv740368190=27, srv714542438=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-17 09:55:04,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-17 09:55:04,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-17 09:55:04,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-17 09:55:04,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-17 09:55:04,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-17 09:55:04,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-17 09:55:04,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-17 09:55:04,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-17 09:55:04,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-17 09:55:04,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-17 09:55:04,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-17 09:55:04,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-17 09:55:04,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-17 09:55:04,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-17 09:55:04,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-17 09:55:04,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-17 09:55:04,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-17 09:55:04,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-17 09:55:04,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-17 09:55:04,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-17 09:55:04,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-17 09:55:04,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-17 09:55:04,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-17 09:55:04,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-17 09:55:04,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-17 09:55:04,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-17 09:55:04,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-17 09:55:04,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-17 09:55:04,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-17 09:55:04,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-17 09:55:04,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-17 09:55:04,061 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-17 09:55:04,061 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1093174527:90 , srv1118631597:90 , srv1192731177:90 , srv1232137032:90 , srv1234533165:90 , srv1259957600:90 , srv1341502263:90 , srv1478243588:90 , srv148413063:90 , srv1489452889:90 , srv1574888719:90 , srv1607296757:90 , srv1678988329:90 , srv1766009553:90 , srv1787914209:90 , srv1861984539:90 , srv1957275895:90 , srv195851430:90 , srv1971247403:90 , srv2093812785:90 , srv2125357509:90 , srv2134640543:90 , srv223953116:90 , srv517355410:90 , srv534218916:90 , srv58975246:90 , srv714542438:90 , srv740368190:90 , srv747080202:90 , srv997832001:90 } 2023-02-17 09:55:04,071 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=191 (was 397), ProcessCount=173 (was 174), AvailableMemoryMB=2430 (was 3186)