2023-07-24 12:14:47,876 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-07-24 12:14:48,296 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-24 12:14:48,406 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-24 12:14:48,480 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-24 12:14:48,491 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-24 12:14:48,505 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=299, ProcessCount=169, AvailableMemoryMB=3064 2023-07-24 12:14:48,507 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-24 12:14:48,508 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-24 12:14:48,670 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv100104828:84 , srv1003093726:84 , srv1004627525:84 , srv1007655986:84 , srv100909276:84 , srv103563285:84 , srv1066176073:84 , srv1070584136:84 , srv1190955903:84 , srv1195232803:84 , srv1206216623:84 , srv1217855846:84 , srv1232762259:84 , srv1245864908:84 , srv1302575640:84 , srv1309627552:84 , srv1339139222:84 , srv1361644582:84 , srv1422700105:84 , srv1423191666:84 , srv1434523497:84 , srv1449097976:84 , srv1462788906:84 , srv1511523770:84 , srv1539330493:84 , srv1551372463:84 , srv1551761810:84 , srv155272289:84 , srv1574119080:84 , srv1602560789:84 , srv1621076757:84 , srv1651256651:84 , srv1668748025:84 , srv1695529014:84 , srv1700335915:84 , srv1729006913:84 , srv1734651883:84 , srv1752966459:84 , srv1798321734:84 , srv1808555094:84 , srv18479357:84 , srv185939887:84 , srv187169224:84 , srv1893674822:84 , srv1896283:84 , srv1898815784:84 , srv1914076569:84 , srv1922518996:84 , srv1953121625:84 , srv1958472773:84 , srv1979204520:84 , srv1990722617:84 , srv2037674560:84 , srv207358152:84 , srv2081255497:84 , srv2100548062:84 , srv2115858713:84 , srv2119709606:84 , srv2143269026:84 , srv243916749:84 , srv296907149:84 , srv323755139:84 , srv355719421:84 , srv372726291:84 , srv387145293:84 , srv423956397:84 , srv504095466:84 , srv508025787:84 , srv530296698:84 , srv560866963:84 , srv563971935:84 , srv578498099:84 , srv581662867:84 , srv58495949:84 , srv612310714:84 , srv616411427:84 , srv619294835:84 , srv652287043:84 , srv656430009:84 , srv683439913:84 , srv696311031:84 , srv713210506:84 , srv71357025:84 , srv729785202:84 , srv738905714:84 , srv743848097:84 , srv746167502:84 , srv780128123:84 , srv781861551:84 , srv810045875:84 , srv810122348:84 , srv851167329:84 , srv858969837:84 , srv879210735:84 , srv902280549:84 , srv913852575:84 , srv913871437:84 , srv951075060:84 , srv955707499:84 , srv1012002803:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-07-24 12:14:48,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-24 12:14:48,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv207358152=54, srv713210506=82, srv1217855846=12, srv2119709606=58, srv1434523497=21, srv746167502=87, srv810122348=91, srv1539330493=25, srv1914076569=47, srv1979204520=51, srv355719421=63, srv1958472773=50, srv743848097=86, srv1206216623=11, srv619294835=77, srv858969837=93, srv781861551=89, srv2037674560=53, srv1003093726=1, srv1232762259=13, srv616411427=76, srv578498099=72, srv2143269026=59, srv1066176073=7, srv955707499=99, srv508025787=68, srv18479357=41, srv851167329=92, srv1893674822=44, srv2115858713=57, srv1990722617=52, srv2081255497=55, srv1070584136=8, srv902280549=95, srv530296698=69, srv652287043=78, srv810045875=90, srv1422700105=19, srv58495949=74, srv913871437=97, srv1195232803=10, srv1361644582=18, srv1734651883=37, srv1012002803=5, srv296907149=61, srv185939887=42, srv1729006913=36, srv504095466=67, srv612310714=75, srv1302575640=15, srv1808555094=40, srv155272289=28, srv1449097976=22, srv100909276=4, srv1898815784=46, srv187169224=43, srv387145293=65, srv729785202=84, srv1798321734=39, srv683439913=80, srv1752966459=38, srv951075060=98, srv103563285=6, srv1551372463=26, srv323755139=62, srv563971935=71, srv780128123=88, srv581662867=73, srv1423191666=20, srv1245864908=14, srv656430009=79, srv1668748025=33, srv1339139222=17, srv1922518996=48, srv372726291=64, srv738905714=85, srv1574119080=29, srv1896283=45, srv1190955903=9, srv423956397=66, srv100104828=0, srv1309627552=16, srv1602560789=30, srv2100548062=56, srv1953121625=49, srv1511523770=24, srv879210735=94, srv1007655986=3, srv560866963=70, srv696311031=81, srv1462788906=23, srv913852575=96, srv1551761810=27, srv1621076757=31, srv1651256651=32, srv1700335915=35, srv1004627525=2, srv1695529014=34, srv71357025=83, srv243916749=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-24 12:14:48,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-24 12:14:48,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-24 12:14:48,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-24 12:14:48,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-24 12:14:48,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-24 12:14:48,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-24 12:14:48,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-24 12:14:48,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-24 12:14:48,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-24 12:14:48,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-24 12:14:48,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-24 12:14:48,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-24 12:14:48,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-24 12:14:48,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-24 12:14:48,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-24 12:14:48,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-24 12:14:48,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-24 12:14:48,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-24 12:14:48,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-24 12:14:48,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-24 12:14:48,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-24 12:14:48,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-24 12:14:48,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-24 12:14:48,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-24 12:14:48,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-24 12:14:48,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-24 12:14:48,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-24 12:14:48,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-24 12:14:48,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-24 12:14:48,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-24 12:14:48,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-24 12:14:48,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-24 12:14:48,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-24 12:14:48,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-24 12:14:48,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-24 12:14:48,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-24 12:14:48,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-24 12:14:48,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-24 12:14:48,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-24 12:14:48,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-24 12:14:48,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-24 12:14:48,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-24 12:14:48,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-24 12:14:48,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-24 12:14:48,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-24 12:14:48,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-24 12:14:48,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-24 12:14:48,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-24 12:14:48,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-24 12:14:48,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-24 12:14:48,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-24 12:14:48,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-24 12:14:48,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-24 12:14:48,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-24 12:14:48,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-24 12:14:48,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-24 12:14:48,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-24 12:14:48,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-24 12:14:48,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-24 12:14:48,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-24 12:14:48,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-24 12:14:48,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-24 12:14:48,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-24 12:14:48,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-24 12:14:48,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-24 12:14:48,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-24 12:14:48,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-24 12:14:48,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-24 12:14:48,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-24 12:14:48,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-24 12:14:48,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-24 12:14:48,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-24 12:14:48,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-24 12:14:48,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-24 12:14:48,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-24 12:14:48,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-24 12:14:48,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-24 12:14:48,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-24 12:14:48,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-24 12:14:48,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-24 12:14:48,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-24 12:14:48,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-24 12:14:48,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-24 12:14:48,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-24 12:14:48,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-24 12:14:48,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-24 12:14:48,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-24 12:14:48,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-24 12:14:48,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-24 12:14:48,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-24 12:14:48,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-24 12:14:48,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-24 12:14:48,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-24 12:14:48,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-24 12:14:48,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-24 12:14:48,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-24 12:14:48,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-24 12:14:48,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-24 12:14:48,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-24 12:14:48,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-24 12:14:48,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-24 12:14:49,040 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-24 12:14:49,041 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-24 12:14:49,041 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-24 12:14:49,451 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 648 ms to try 5000 different iterations. Found a solution that moves 4661 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1863562432863647. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04062057577087136, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05573707718645139, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5178888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04062057577087136, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18175, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24716666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-24 12:14:49,469 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1309627552:82 , srv1574119080:82 , srv1896283:82 , srv2100548062:82 , srv1339139222:83 , srv1551761810:83 , srv243916749:83 , srv504095466:83 , srv913852575:83 , srv100104828:84 , srv1066176073:84 , srv1511523770:84 , srv1551372463:84 , srv1668748025:84 , srv1729006913:84 , srv18479357:84 , srv185939887:84 , srv1953121625:84 , srv207358152:84 , srv2143269026:84 , srv372726291:84 , srv578498099:84 , srv652287043:84 , srv746167502:84 , srv780128123:84 , srv1003093726:85 , srv1007655986:85 , srv1245864908:85 , srv1302575640:85 , srv1449097976:85 , srv1700335915:85 , srv1898815784:85 , srv1914076569:85 , srv1958472773:85 , srv2115858713:85 , srv581662867:85 , srv696311031:85 , srv713210506:85 , srv729785202:85 , srv858969837:85 , srv951075060:85 , srv1195232803:86 , srv1217855846:86 , srv1232762259:86 , srv1423191666:86 , srv1602560789:86 , srv1695529014:86 , srv1893674822:86 , srv2119709606:86 , srv296907149:86 , srv387145293:86 , srv560866963:86 , srv616411427:86 , srv619294835:86 , srv71357025:86 , srv743848097:86 , srv781861551:86 , srv810045875:86 , srv103563285:87 , srv1070584136:87 , srv1190955903:87 , srv1206216623:87 , srv1462788906:87 , srv155272289:87 , srv1651256651:87 , srv1798321734:87 , srv1808555094:87 , srv1922518996:87 , srv1979204520:87 , srv1990722617:87 , srv2037674560:87 , srv423956397:87 , srv508025787:87 , srv902280549:87 , srv100909276:88 , srv1422700105:88 , srv1434523497:88 , srv323755139:88 , srv530296698:88 , srv58495949:88 , srv612310714:88 , srv656430009:88 , srv1361644582:89 , srv1539330493:89 , srv879210735:89 , srv913871437:89 , srv1004627525:90 , srv1621076757:90 , srv1752966459:90 , srv187169224:90 , srv355719421:90 , srv738905714:90 , srv2081255497:91 , srv683439913:91 , srv955707499:91 , srv810122348:92 , srv563971935:93 , srv1734651883:95 , srv851167329:95 , srv1012002803:451 } 2023-07-24 12:14:49,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-24 12:14:49,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv207358152=54, srv713210506=82, srv1217855846=12, srv2119709606=58, srv1434523497=21, srv746167502=87, srv810122348=91, srv1539330493=25, srv1914076569=47, srv1979204520=51, srv355719421=63, srv1958472773=50, srv743848097=86, srv1206216623=11, srv619294835=77, srv858969837=93, srv781861551=89, srv2037674560=53, srv1003093726=1, srv1232762259=13, srv616411427=76, srv578498099=72, srv2143269026=59, srv1066176073=7, srv955707499=99, srv508025787=68, srv18479357=41, srv851167329=92, srv1893674822=44, srv2115858713=57, srv1990722617=52, srv2081255497=55, srv1070584136=8, srv902280549=95, srv530296698=69, srv652287043=78, srv810045875=90, srv1422700105=19, srv58495949=74, srv913871437=97, srv1195232803=10, srv1361644582=18, srv1734651883=37, srv1012002803=5, srv296907149=61, srv185939887=42, srv1729006913=36, srv504095466=67, srv612310714=75, srv1302575640=15, srv1808555094=40, srv155272289=28, srv1449097976=22, srv100909276=4, srv1898815784=46, srv187169224=43, srv387145293=65, srv729785202=84, srv1798321734=39, srv683439913=80, srv1752966459=38, srv951075060=98, srv103563285=6, srv1551372463=26, srv323755139=62, srv563971935=71, srv780128123=88, srv581662867=73, srv1423191666=20, srv1245864908=14, srv656430009=79, srv1668748025=33, srv1339139222=17, srv1922518996=48, srv372726291=64, srv738905714=85, srv1574119080=29, srv1896283=45, srv1190955903=9, srv423956397=66, srv100104828=0, srv1309627552=16, srv1602560789=30, srv2100548062=56, srv1953121625=49, srv1511523770=24, srv879210735=94, srv1007655986=3, srv560866963=70, srv696311031=81, srv1462788906=23, srv913852575=96, srv1551761810=27, srv1621076757=31, srv1651256651=32, srv1700335915=35, srv1004627525=2, srv1695529014=34, srv71357025=83, srv243916749=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-24 12:14:49,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-24 12:14:49,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-24 12:14:49,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-24 12:14:49,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-24 12:14:49,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-24 12:14:49,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-24 12:14:49,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-24 12:14:49,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-24 12:14:49,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-24 12:14:49,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-24 12:14:49,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-24 12:14:49,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-24 12:14:49,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-24 12:14:49,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-24 12:14:49,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-24 12:14:49,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-24 12:14:49,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-24 12:14:49,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-24 12:14:49,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-24 12:14:49,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-24 12:14:49,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-24 12:14:49,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-24 12:14:49,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-24 12:14:49,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-24 12:14:49,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-24 12:14:49,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-24 12:14:49,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-24 12:14:49,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-24 12:14:49,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-24 12:14:49,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-24 12:14:49,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-24 12:14:49,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-24 12:14:49,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-24 12:14:49,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-24 12:14:49,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-24 12:14:49,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-24 12:14:49,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-24 12:14:49,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-24 12:14:49,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-24 12:14:49,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-24 12:14:49,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-24 12:14:49,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-24 12:14:49,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-24 12:14:49,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-24 12:14:49,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-24 12:14:49,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-24 12:14:49,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-24 12:14:49,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-24 12:14:49,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-24 12:14:49,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-24 12:14:49,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-24 12:14:49,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-24 12:14:49,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-24 12:14:49,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-24 12:14:49,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-24 12:14:49,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-24 12:14:49,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-24 12:14:49,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-24 12:14:49,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-24 12:14:49,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-24 12:14:49,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-24 12:14:49,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-24 12:14:49,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-24 12:14:49,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-24 12:14:49,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-24 12:14:49,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-24 12:14:49,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-24 12:14:49,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-24 12:14:49,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-24 12:14:49,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-24 12:14:49,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-24 12:14:49,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-24 12:14:49,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-24 12:14:49,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-24 12:14:49,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-24 12:14:49,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-24 12:14:49,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-24 12:14:49,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-24 12:14:49,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-24 12:14:49,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-24 12:14:49,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-24 12:14:49,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-24 12:14:49,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-24 12:14:49,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-24 12:14:49,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-24 12:14:49,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-24 12:14:49,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-24 12:14:49,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-24 12:14:49,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-24 12:14:49,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-24 12:14:49,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-24 12:14:49,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-24 12:14:49,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-24 12:14:49,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-24 12:14:49,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-24 12:14:49,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-24 12:14:49,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-24 12:14:49,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-24 12:14:49,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-24 12:14:49,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-24 12:14:49,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-24 12:14:49,702 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04062057577087136, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05573707718645139, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04062057577087136, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18175, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24716666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-24 12:14:49,702 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-24 12:14:49,702 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1863236062667997, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04062057577087136, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05573707718645139, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04062057577087136, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18175, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24716666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-24 12:14:50,054 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 504 ms to try 5000 different iterations. Found a solution that moves 2667 regions; Going from a computed imbalance of 0.1863236062667997 to a new imbalance of 0.01402416454806252. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022496622351124807); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03563753176124393, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29633333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022496622351124807); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-24 12:14:50,078 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1734651883:78 , srv851167329:78 , srv563971935:79 , srv187169224:80 , srv2081255497:80 , srv683439913:80 , srv1752966459:81 , srv1004627525:82 , srv1539330493:82 , srv1621076757:82 , srv810122348:82 , srv879210735:82 , srv913871437:82 , srv955707499:82 , srv1434523497:83 , srv1651256651:83 , srv1979204520:83 , srv530296698:83 , srv1190955903:84 , srv1206216623:84 , srv1217855846:84 , srv1422700105:84 , srv1462788906:84 , srv1798321734:84 , srv323755139:84 , srv387145293:84 , srv508025787:84 , srv612310714:84 , srv619294835:84 , srv656430009:84 , srv103563285:85 , srv1070584136:85 , srv1232762259:85 , srv1361644582:85 , srv155272289:85 , srv1602560789:85 , srv1808555094:85 , srv1990722617:85 , srv355719421:85 , srv423956397:85 , srv560866963:85 , srv58495949:85 , srv738905714:85 , srv743848097:85 , srv781861551:85 , srv810045875:85 , srv1007655986:86 , srv1423191666:86 , srv1551372463:86 , srv1893674822:86 , srv1914076569:86 , srv1922518996:86 , srv1958472773:86 , srv2037674560:86 , srv581662867:86 , srv616411427:86 , srv696311031:86 , srv71357025:86 , srv100909276:87 , srv1066176073:87 , srv1302575640:87 , srv1339139222:87 , srv1695529014:87 , srv1700335915:87 , srv1729006913:87 , srv1953121625:87 , srv207358152:87 , srv2100548062:87 , srv2115858713:87 , srv2119709606:87 , srv578498099:87 , srv652287043:87 , srv713210506:87 , srv902280549:87 , srv100104828:88 , srv1003093726:88 , srv185939887:88 , srv2143269026:88 , srv296907149:88 , srv729785202:88 , srv746167502:88 , srv780128123:88 , srv858969837:88 , srv913852575:88 , srv951075060:88 , srv1309627552:89 , srv1449097976:89 , srv1511523770:89 , srv1574119080:89 , srv18479357:89 , srv243916749:89 , srv372726291:89 , srv1195232803:90 , srv1245864908:90 , srv1668748025:90 , srv1898815784:90 , srv504095466:90 , srv1551761810:91 , srv1896283:91 , srv1012002803:523 } 2023-07-24 12:14:50,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-24 12:14:50,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv207358152=54, srv713210506=82, srv1217855846=12, srv2119709606=58, srv1434523497=21, srv746167502=87, srv810122348=91, srv1539330493=25, srv1914076569=47, srv1979204520=51, srv355719421=63, srv1958472773=50, srv743848097=86, srv1206216623=11, srv619294835=77, srv858969837=93, srv781861551=89, srv2037674560=53, srv1003093726=1, srv1232762259=13, srv616411427=76, srv578498099=72, srv2143269026=59, srv1066176073=7, srv955707499=99, srv508025787=68, srv18479357=41, srv851167329=92, srv1893674822=44, srv2115858713=57, srv1990722617=52, srv2081255497=55, srv1070584136=8, srv902280549=95, srv530296698=69, srv652287043=78, srv810045875=90, srv1422700105=19, srv58495949=74, srv913871437=97, srv1195232803=10, srv1361644582=18, srv1734651883=37, srv1012002803=5, srv296907149=61, srv185939887=42, srv1729006913=36, srv504095466=67, srv612310714=75, srv1302575640=15, srv1808555094=40, srv155272289=28, srv1449097976=22, srv100909276=4, srv1898815784=46, srv187169224=43, srv387145293=65, srv729785202=84, srv1798321734=39, srv683439913=80, srv1752966459=38, srv951075060=98, srv103563285=6, srv1551372463=26, srv323755139=62, srv563971935=71, srv780128123=88, srv581662867=73, srv1423191666=20, srv1245864908=14, srv656430009=79, srv1668748025=33, srv1339139222=17, srv1922518996=48, srv372726291=64, srv738905714=85, srv1574119080=29, srv1896283=45, srv1190955903=9, srv423956397=66, srv100104828=0, srv1309627552=16, srv1602560789=30, srv2100548062=56, srv1953121625=49, srv1511523770=24, srv879210735=94, srv1007655986=3, srv560866963=70, srv696311031=81, srv1462788906=23, srv913852575=96, srv1551761810=27, srv1621076757=31, srv1651256651=32, srv1700335915=35, srv1004627525=2, srv1695529014=34, srv71357025=83, srv243916749=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-24 12:14:50,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-24 12:14:50,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-24 12:14:50,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-24 12:14:50,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-24 12:14:50,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-24 12:14:50,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-24 12:14:50,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-24 12:14:50,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-24 12:14:50,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-24 12:14:50,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-24 12:14:50,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-24 12:14:50,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-24 12:14:50,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-24 12:14:50,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-24 12:14:50,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-24 12:14:50,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-24 12:14:50,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-24 12:14:50,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-24 12:14:50,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-24 12:14:50,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-24 12:14:50,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-24 12:14:50,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-24 12:14:50,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-24 12:14:50,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-24 12:14:50,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-24 12:14:50,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-24 12:14:50,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-24 12:14:50,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-24 12:14:50,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-24 12:14:50,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-24 12:14:50,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-24 12:14:50,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-24 12:14:50,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-24 12:14:50,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-24 12:14:50,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-24 12:14:50,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-24 12:14:50,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-24 12:14:50,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-24 12:14:50,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-24 12:14:50,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-24 12:14:50,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-24 12:14:50,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-24 12:14:50,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-24 12:14:50,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-24 12:14:50,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-24 12:14:50,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-24 12:14:50,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-24 12:14:50,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-24 12:14:50,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-24 12:14:50,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-24 12:14:50,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-24 12:14:50,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-24 12:14:50,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-24 12:14:50,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-24 12:14:50,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-24 12:14:50,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-24 12:14:50,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-24 12:14:50,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-24 12:14:50,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-24 12:14:50,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-24 12:14:50,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-24 12:14:50,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-24 12:14:50,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-24 12:14:50,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-24 12:14:50,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-24 12:14:50,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-24 12:14:50,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-24 12:14:50,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-24 12:14:50,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-24 12:14:50,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-24 12:14:50,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-24 12:14:50,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-24 12:14:50,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-24 12:14:50,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-24 12:14:50,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-24 12:14:50,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-24 12:14:50,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-24 12:14:50,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-24 12:14:50,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-24 12:14:50,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-24 12:14:50,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-24 12:14:50,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-24 12:14:50,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-24 12:14:50,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-24 12:14:50,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-24 12:14:50,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-24 12:14:50,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-24 12:14:50,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-24 12:14:50,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-24 12:14:50,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-24 12:14:50,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-24 12:14:50,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-24 12:14:50,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-24 12:14:50,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-24 12:14:50,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-24 12:14:50,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-24 12:14:50,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-24 12:14:50,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-24 12:14:50,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-24 12:14:50,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-24 12:14:50,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-24 12:14:50,264 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022496622351124807); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03563753176124393, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022496622351124807); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-24 12:14:50,264 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-24 12:14:50,265 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014005489814919442, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022496622351124807); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03563753176124393, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022496622351124807); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-24 12:14:50,541 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 391 ms to try 5000 different iterations. Found a solution that moves 1009 regions; Going from a computed imbalance of 0.014005489814919442 to a new imbalance of 6.604880447102291E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0077769760689906375); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010172474468514046); MoveCostFunction : (multiplier=7.0, imbalance=0.1121111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0077769760689906375); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-24 12:14:50,545 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1195232803:81 , srv100909276:82 , srv1898815784:82 , srv738905714:82 , srv1511523770:83 , srv1668748025:83 , srv355719421:83 , srv1003093726:84 , srv1245864908:84 , srv1423191666:84 , srv1449097976:84 , srv1551761810:84 , srv1695529014:84 , srv1700335915:84 , srv18479357:84 , srv1896283:84 , srv1922518996:84 , srv2037674560:84 , srv296907149:84 , srv372726291:84 , srv504095466:84 , srv58495949:84 , srv71357025:84 , srv729785202:84 , srv851167329:84 , srv858969837:84 , srv951075060:84 , srv955707499:84 , srv100104828:85 , srv1004627525:85 , srv103563285:85 , srv1302575640:85 , srv1361644582:85 , srv1422700105:85 , srv155272289:85 , srv1734651883:85 , srv1808555094:85 , srv185939887:85 , srv2115858713:85 , srv2119709606:85 , srv243916749:85 , srv323755139:85 , srv563971935:85 , srv683439913:85 , srv746167502:85 , srv780128123:85 , srv810122348:85 , srv902280549:85 , srv1066176073:86 , srv1070584136:86 , srv1206216623:86 , srv1232762259:86 , srv1309627552:86 , srv1462788906:86 , srv1539330493:86 , srv1574119080:86 , srv1602560789:86 , srv1621076757:86 , srv1729006913:86 , srv1752966459:86 , srv1798321734:86 , srv1893674822:86 , srv1914076569:86 , srv1953121625:86 , srv1990722617:86 , srv207358152:86 , srv2081255497:86 , srv2143269026:86 , srv423956397:86 , srv508025787:86 , srv530296698:86 , srv560866963:86 , srv581662867:86 , srv612310714:86 , srv616411427:86 , srv652287043:86 , srv656430009:86 , srv696311031:86 , srv713210506:86 , srv913871437:86 , srv1007655986:87 , srv1190955903:87 , srv1217855846:87 , srv1339139222:87 , srv1434523497:87 , srv1651256651:87 , srv187169224:87 , srv1958472773:87 , srv2100548062:87 , srv387145293:87 , srv578498099:87 , srv619294835:87 , srv743848097:87 , srv781861551:87 , srv810045875:87 , srv879210735:87 , srv913852575:87 , srv1551372463:88 , srv1979204520:88 , srv1012002803:553 } 2023-07-24 12:14:50,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-24 12:14:50,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv207358152=54, srv713210506=82, srv1217855846=12, srv2119709606=58, srv1434523497=21, srv746167502=87, srv810122348=91, srv1539330493=25, srv1914076569=47, srv1979204520=51, srv355719421=63, srv1958472773=50, srv743848097=86, srv1206216623=11, srv619294835=77, srv858969837=93, srv781861551=89, srv2037674560=53, srv1003093726=1, srv1232762259=13, srv616411427=76, srv578498099=72, srv2143269026=59, srv1066176073=7, srv955707499=99, srv508025787=68, srv18479357=41, srv851167329=92, srv1893674822=44, srv2115858713=57, srv1990722617=52, srv2081255497=55, srv1070584136=8, srv902280549=95, srv530296698=69, srv652287043=78, srv810045875=90, srv1422700105=19, srv58495949=74, srv913871437=97, srv1195232803=10, srv1361644582=18, srv1734651883=37, srv1012002803=5, srv296907149=61, srv185939887=42, srv1729006913=36, srv504095466=67, srv612310714=75, srv1302575640=15, srv1808555094=40, srv155272289=28, srv1449097976=22, srv100909276=4, srv1898815784=46, srv187169224=43, srv387145293=65, srv729785202=84, srv1798321734=39, srv683439913=80, srv1752966459=38, srv951075060=98, srv103563285=6, srv1551372463=26, srv323755139=62, srv563971935=71, srv780128123=88, srv581662867=73, srv1423191666=20, srv1245864908=14, srv656430009=79, srv1668748025=33, srv1339139222=17, srv1922518996=48, srv372726291=64, srv738905714=85, srv1574119080=29, srv1896283=45, srv1190955903=9, srv423956397=66, srv100104828=0, srv1309627552=16, srv1602560789=30, srv2100548062=56, srv1953121625=49, srv1511523770=24, srv879210735=94, srv1007655986=3, srv560866963=70, srv696311031=81, srv1462788906=23, srv913852575=96, srv1551761810=27, srv1621076757=31, srv1651256651=32, srv1700335915=35, srv1004627525=2, srv1695529014=34, srv71357025=83, srv243916749=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-24 12:14:50,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-24 12:14:50,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-24 12:14:50,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-24 12:14:50,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-24 12:14:50,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-24 12:14:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-24 12:14:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-24 12:14:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-24 12:14:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-24 12:14:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-24 12:14:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-24 12:14:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-24 12:14:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-24 12:14:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-24 12:14:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-24 12:14:50,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-24 12:14:50,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-24 12:14:50,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-24 12:14:50,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-24 12:14:50,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-24 12:14:50,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-24 12:14:50,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-24 12:14:50,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-24 12:14:50,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-24 12:14:50,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-24 12:14:50,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-24 12:14:50,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-24 12:14:50,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-24 12:14:50,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-24 12:14:50,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-24 12:14:50,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-24 12:14:50,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-24 12:14:50,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-24 12:14:50,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-24 12:14:50,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-24 12:14:50,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-24 12:14:50,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-24 12:14:50,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-24 12:14:50,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-24 12:14:50,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-24 12:14:50,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-24 12:14:50,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-24 12:14:50,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-24 12:14:50,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-24 12:14:50,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-24 12:14:50,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-24 12:14:50,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-24 12:14:50,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-24 12:14:50,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-24 12:14:50,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-24 12:14:50,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-24 12:14:50,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-24 12:14:50,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-24 12:14:50,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-24 12:14:50,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-24 12:14:50,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-24 12:14:50,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-24 12:14:50,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-24 12:14:50,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-24 12:14:50,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-24 12:14:50,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-24 12:14:50,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-24 12:14:50,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-24 12:14:50,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-24 12:14:50,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-24 12:14:50,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-24 12:14:50,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-24 12:14:50,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-24 12:14:50,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-24 12:14:50,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-24 12:14:50,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-24 12:14:50,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-24 12:14:50,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-24 12:14:50,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-24 12:14:50,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-24 12:14:50,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-24 12:14:50,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-24 12:14:50,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-24 12:14:50,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-24 12:14:50,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-24 12:14:50,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-24 12:14:50,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-24 12:14:50,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-24 12:14:50,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-24 12:14:50,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-24 12:14:50,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-24 12:14:50,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-24 12:14:50,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-24 12:14:50,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-24 12:14:50,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-24 12:14:50,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-24 12:14:50,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-24 12:14:50,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-24 12:14:50,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-24 12:14:50,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-24 12:14:50,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-24 12:14:50,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-24 12:14:50,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-24 12:14:50,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-24 12:14:50,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-24 12:14:50,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-24 12:14:50,737 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0077769760689906375); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010172474468514046); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0077769760689906375); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-24 12:14:50,738 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-24 12:14:50,738 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.53422875703344E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0077769760689906375); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010172474468514046); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0077769760689906375); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-24 12:14:51,202 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 607 ms to try 5000 different iterations. Found a solution that moves 273 regions; Going from a computed imbalance of 6.53422875703344E-4 to a new imbalance of 2.757258643267612E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.030333333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-24 12:14:51,203 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1434523497:83 , srv2143269026:83 , srv810122348:83 , srv902280549:83 , srv1007655986:84 , srv1070584136:84 , srv1190955903:84 , srv1245864908:84 , srv1361644582:84 , srv1449097976:84 , srv1551372463:84 , srv1551761810:84 , srv1621076757:84 , srv1695529014:84 , srv1734651883:84 , srv1893674822:84 , srv1953121625:84 , srv1958472773:84 , srv1979204520:84 , srv1990722617:84 , srv2081255497:84 , srv2119709606:84 , srv296907149:84 , srv355719421:84 , srv423956397:84 , srv578498099:84 , srv612310714:84 , srv616411427:84 , srv656430009:84 , srv713210506:84 , srv729785202:84 , srv743848097:84 , srv781861551:84 , srv913852575:84 , srv100104828:85 , srv1003093726:85 , srv1004627525:85 , srv100909276:85 , srv1066176073:85 , srv1195232803:85 , srv1206216623:85 , srv1217855846:85 , srv1232762259:85 , srv1302575640:85 , srv1309627552:85 , srv1339139222:85 , srv1422700105:85 , srv1462788906:85 , srv1539330493:85 , srv155272289:85 , srv1574119080:85 , srv1602560789:85 , srv1651256651:85 , srv1668748025:85 , srv1729006913:85 , srv1752966459:85 , srv1798321734:85 , srv1808555094:85 , srv18479357:85 , srv187169224:85 , srv1896283:85 , srv1898815784:85 , srv1914076569:85 , srv1922518996:85 , srv2037674560:85 , srv207358152:85 , srv2115858713:85 , srv243916749:85 , srv323755139:85 , srv372726291:85 , srv387145293:85 , srv504095466:85 , srv508025787:85 , srv530296698:85 , srv560866963:85 , srv563971935:85 , srv581662867:85 , srv58495949:85 , srv619294835:85 , srv652287043:85 , srv696311031:85 , srv738905714:85 , srv746167502:85 , srv780128123:85 , srv810045875:85 , srv851167329:85 , srv858969837:85 , srv879210735:85 , srv913871437:85 , srv951075060:85 , srv955707499:85 , srv103563285:86 , srv1423191666:86 , srv1511523770:86 , srv1700335915:86 , srv185939887:86 , srv2100548062:86 , srv683439913:86 , srv71357025:86 , srv1012002803:615 } 2023-07-24 12:14:51,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-24 12:14:51,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv207358152=54, srv713210506=82, srv1217855846=12, srv2119709606=58, srv1434523497=21, srv746167502=87, srv810122348=91, srv1539330493=25, srv1914076569=47, srv1979204520=51, srv355719421=63, srv1958472773=50, srv743848097=86, srv1206216623=11, srv619294835=77, srv858969837=93, srv781861551=89, srv2037674560=53, srv1003093726=1, srv1232762259=13, srv616411427=76, srv578498099=72, srv2143269026=59, srv1066176073=7, srv955707499=99, srv508025787=68, srv18479357=41, srv851167329=92, srv1893674822=44, srv2115858713=57, srv1990722617=52, srv2081255497=55, srv1070584136=8, srv902280549=95, srv530296698=69, srv652287043=78, srv810045875=90, srv1422700105=19, srv58495949=74, srv913871437=97, srv1195232803=10, srv1361644582=18, srv1734651883=37, srv1012002803=5, srv296907149=61, srv185939887=42, srv1729006913=36, srv504095466=67, srv612310714=75, srv1302575640=15, srv1808555094=40, srv155272289=28, srv1449097976=22, srv100909276=4, srv1898815784=46, srv187169224=43, srv387145293=65, srv729785202=84, srv1798321734=39, srv683439913=80, srv1752966459=38, srv951075060=98, srv103563285=6, srv1551372463=26, srv323755139=62, srv563971935=71, srv780128123=88, srv581662867=73, srv1423191666=20, srv1245864908=14, srv656430009=79, srv1668748025=33, srv1339139222=17, srv1922518996=48, srv372726291=64, srv738905714=85, srv1574119080=29, srv1896283=45, srv1190955903=9, srv423956397=66, srv100104828=0, srv1309627552=16, srv1602560789=30, srv2100548062=56, srv1953121625=49, srv1511523770=24, srv879210735=94, srv1007655986=3, srv560866963=70, srv696311031=81, srv1462788906=23, srv913852575=96, srv1551761810=27, srv1621076757=31, srv1651256651=32, srv1700335915=35, srv1004627525=2, srv1695529014=34, srv71357025=83, srv243916749=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-24 12:14:51,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-24 12:14:51,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-24 12:14:51,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-24 12:14:51,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-24 12:14:51,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-24 12:14:51,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-24 12:14:51,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-24 12:14:51,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-24 12:14:51,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-24 12:14:51,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-24 12:14:51,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-24 12:14:51,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-24 12:14:51,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-24 12:14:51,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-24 12:14:51,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-24 12:14:51,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-24 12:14:51,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-24 12:14:51,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-24 12:14:51,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-24 12:14:51,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-24 12:14:51,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-24 12:14:51,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-24 12:14:51,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-24 12:14:51,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-24 12:14:51,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-24 12:14:51,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-24 12:14:51,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-24 12:14:51,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-24 12:14:51,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-24 12:14:51,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-24 12:14:51,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-24 12:14:51,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-24 12:14:51,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-24 12:14:51,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-24 12:14:51,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-24 12:14:51,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-24 12:14:51,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-24 12:14:51,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-24 12:14:51,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-24 12:14:51,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-24 12:14:51,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-24 12:14:51,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-24 12:14:51,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-24 12:14:51,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-24 12:14:51,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-24 12:14:51,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-24 12:14:51,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-24 12:14:51,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-24 12:14:51,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-24 12:14:51,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-24 12:14:51,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-24 12:14:51,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-24 12:14:51,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-24 12:14:51,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-24 12:14:51,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-24 12:14:51,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-24 12:14:51,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-24 12:14:51,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-24 12:14:51,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-24 12:14:51,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-24 12:14:51,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-24 12:14:51,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-24 12:14:51,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-24 12:14:51,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-24 12:14:51,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-24 12:14:51,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-24 12:14:51,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-24 12:14:51,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-24 12:14:51,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-24 12:14:51,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-24 12:14:51,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-24 12:14:51,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-24 12:14:51,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-24 12:14:51,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-24 12:14:51,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-24 12:14:51,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-24 12:14:51,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-24 12:14:51,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-24 12:14:51,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-24 12:14:51,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-24 12:14:51,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-24 12:14:51,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-24 12:14:51,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-24 12:14:51,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-24 12:14:51,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-24 12:14:51,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-24 12:14:51,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-24 12:14:51,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-24 12:14:51,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-24 12:14:51,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-24 12:14:51,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-24 12:14:51,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-24 12:14:51,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-24 12:14:51,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-24 12:14:51,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-24 12:14:51,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-24 12:14:51,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-24 12:14:51,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-24 12:14:51,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-24 12:14:51,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-24 12:14:51,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-24 12:14:51,408 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-24 12:14:51,408 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-24 12:14:51,409 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.566099957551097E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-24 12:14:51,720 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 468 ms to try 5000 different iterations. Found a solution that moves 21 regions; Going from a computed imbalance of 2.566099957551097E-5 to a new imbalance of 7.220284439188305E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0023333333333333335); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-24 12:14:51,721 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv103563285:83 , srv1245864908:83 , srv1361644582:83 , srv185939887:83 , srv810045875:83 , srv879210735:83 , srv100104828:84 , srv1003093726:84 , srv1004627525:84 , srv1007655986:84 , srv100909276:84 , srv1066176073:84 , srv1070584136:84 , srv1190955903:84 , srv1195232803:84 , srv1206216623:84 , srv1217855846:84 , srv1232762259:84 , srv1302575640:84 , srv1309627552:84 , srv1339139222:84 , srv1422700105:84 , srv1423191666:84 , srv1449097976:84 , srv1462788906:84 , srv1511523770:84 , srv1539330493:84 , srv1551372463:84 , srv1551761810:84 , srv155272289:84 , srv1574119080:84 , srv1602560789:84 , srv1621076757:84 , srv1651256651:84 , srv1668748025:84 , srv1700335915:84 , srv1729006913:84 , srv1734651883:84 , srv1752966459:84 , srv1798321734:84 , srv1808555094:84 , srv18479357:84 , srv187169224:84 , srv1893674822:84 , srv1896283:84 , srv1898815784:84 , srv1914076569:84 , srv1922518996:84 , srv1958472773:84 , srv1979204520:84 , srv1990722617:84 , srv2037674560:84 , srv207358152:84 , srv2100548062:84 , srv2115858713:84 , srv2119709606:84 , srv243916749:84 , srv296907149:84 , srv323755139:84 , srv355719421:84 , srv372726291:84 , srv387145293:84 , srv423956397:84 , srv504095466:84 , srv508025787:84 , srv530296698:84 , srv560866963:84 , srv563971935:84 , srv578498099:84 , srv581662867:84 , srv58495949:84 , srv612310714:84 , srv616411427:84 , srv619294835:84 , srv652287043:84 , srv656430009:84 , srv683439913:84 , srv696311031:84 , srv713210506:84 , srv71357025:84 , srv738905714:84 , srv743848097:84 , srv746167502:84 , srv780128123:84 , srv781861551:84 , srv810122348:84 , srv851167329:84 , srv858969837:84 , srv902280549:84 , srv913852575:84 , srv913871437:84 , srv951075060:84 , srv955707499:84 , srv1434523497:85 , srv1695529014:85 , srv1953121625:85 , srv2081255497:85 , srv2143269026:85 , srv729785202:85 , srv1012002803:684 } 2023-07-24 12:14:51,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-24 12:14:51,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv207358152=54, srv713210506=82, srv1217855846=12, srv2119709606=58, srv1434523497=21, srv746167502=87, srv810122348=91, srv1539330493=25, srv1914076569=47, srv1979204520=51, srv355719421=63, srv1958472773=50, srv743848097=86, srv1206216623=11, srv619294835=77, srv858969837=93, srv781861551=89, srv2037674560=53, srv1003093726=1, srv1232762259=13, srv616411427=76, srv578498099=72, srv2143269026=59, srv1066176073=7, srv955707499=99, srv508025787=68, srv18479357=41, srv851167329=92, srv1893674822=44, srv2115858713=57, srv1990722617=52, srv2081255497=55, srv1070584136=8, srv902280549=95, srv530296698=69, srv652287043=78, srv810045875=90, srv1422700105=19, srv58495949=74, srv913871437=97, srv1195232803=10, srv1361644582=18, srv1734651883=37, srv1012002803=5, srv296907149=61, srv185939887=42, srv1729006913=36, srv504095466=67, srv612310714=75, srv1302575640=15, srv1808555094=40, srv155272289=28, srv1449097976=22, srv100909276=4, srv1898815784=46, srv187169224=43, srv387145293=65, srv729785202=84, srv1798321734=39, srv683439913=80, srv1752966459=38, srv951075060=98, srv103563285=6, srv1551372463=26, srv323755139=62, srv563971935=71, srv780128123=88, srv581662867=73, srv1423191666=20, srv1245864908=14, srv656430009=79, srv1668748025=33, srv1339139222=17, srv1922518996=48, srv372726291=64, srv738905714=85, srv1574119080=29, srv1896283=45, srv1190955903=9, srv423956397=66, srv100104828=0, srv1309627552=16, srv1602560789=30, srv2100548062=56, srv1953121625=49, srv1511523770=24, srv879210735=94, srv1007655986=3, srv560866963=70, srv696311031=81, srv1462788906=23, srv913852575=96, srv1551761810=27, srv1621076757=31, srv1651256651=32, srv1700335915=35, srv1004627525=2, srv1695529014=34, srv71357025=83, srv243916749=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-24 12:14:51,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-24 12:14:51,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-24 12:14:51,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-24 12:14:51,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-24 12:14:51,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-24 12:14:51,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-24 12:14:51,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-24 12:14:51,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-24 12:14:51,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-24 12:14:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-24 12:14:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-24 12:14:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-24 12:14:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-24 12:14:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-24 12:14:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-24 12:14:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-24 12:14:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-24 12:14:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-24 12:14:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-24 12:14:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-24 12:14:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-24 12:14:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-24 12:14:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-24 12:14:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-24 12:14:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-24 12:14:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-24 12:14:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-24 12:14:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-24 12:14:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-24 12:14:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-24 12:14:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-24 12:14:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-24 12:14:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-24 12:14:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-24 12:14:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-24 12:14:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-24 12:14:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-24 12:14:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-24 12:14:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-24 12:14:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-24 12:14:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-24 12:14:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-24 12:14:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-24 12:14:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-24 12:14:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-24 12:14:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-24 12:14:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-24 12:14:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-24 12:14:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-24 12:14:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-24 12:14:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-24 12:14:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-24 12:14:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-24 12:14:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-24 12:14:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-24 12:14:51,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-24 12:14:51,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-24 12:14:51,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-24 12:14:51,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-24 12:14:51,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-24 12:14:51,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-24 12:14:51,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-24 12:14:51,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-24 12:14:51,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-24 12:14:51,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-24 12:14:51,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-24 12:14:51,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-24 12:14:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-24 12:14:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-24 12:14:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-24 12:14:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-24 12:14:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-24 12:14:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-24 12:14:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-24 12:14:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-24 12:14:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-24 12:14:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-24 12:14:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-24 12:14:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-24 12:14:51,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-24 12:14:51,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-24 12:14:51,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-24 12:14:51,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-24 12:14:51,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-24 12:14:51,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-24 12:14:51,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-24 12:14:51,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-24 12:14:51,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-24 12:14:51,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-24 12:14:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-24 12:14:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-24 12:14:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-24 12:14:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-24 12:14:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-24 12:14:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-24 12:14:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-24 12:14:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-24 12:14:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-24 12:14:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-24 12:14:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-24 12:14:51,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-24 12:14:51,882 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-24 12:14:51,882 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv103563285:83 , srv1245864908:83 , srv1361644582:83 , srv185939887:83 , srv810045875:83 , srv879210735:83 , srv100104828:84 , srv1003093726:84 , srv1004627525:84 , srv1007655986:84 , srv100909276:84 , srv1066176073:84 , srv1070584136:84 , srv1190955903:84 , srv1195232803:84 , srv1206216623:84 , srv1217855846:84 , srv1232762259:84 , srv1302575640:84 , srv1309627552:84 , srv1339139222:84 , srv1422700105:84 , srv1423191666:84 , srv1449097976:84 , srv1462788906:84 , srv1511523770:84 , srv1539330493:84 , srv1551372463:84 , srv1551761810:84 , srv155272289:84 , srv1574119080:84 , srv1602560789:84 , srv1621076757:84 , srv1651256651:84 , srv1668748025:84 , srv1700335915:84 , srv1729006913:84 , srv1734651883:84 , srv1752966459:84 , srv1798321734:84 , srv1808555094:84 , srv18479357:84 , srv187169224:84 , srv1893674822:84 , srv1896283:84 , srv1898815784:84 , srv1914076569:84 , srv1922518996:84 , srv1958472773:84 , srv1979204520:84 , srv1990722617:84 , srv2037674560:84 , srv207358152:84 , srv2100548062:84 , srv2115858713:84 , srv2119709606:84 , srv243916749:84 , srv296907149:84 , srv323755139:84 , srv355719421:84 , srv372726291:84 , srv387145293:84 , srv423956397:84 , srv504095466:84 , srv508025787:84 , srv530296698:84 , srv560866963:84 , srv563971935:84 , srv578498099:84 , srv581662867:84 , srv58495949:84 , srv612310714:84 , srv616411427:84 , srv619294835:84 , srv652287043:84 , srv656430009:84 , srv683439913:84 , srv696311031:84 , srv713210506:84 , srv71357025:84 , srv738905714:84 , srv743848097:84 , srv746167502:84 , srv780128123:84 , srv781861551:84 , srv810122348:84 , srv851167329:84 , srv858969837:84 , srv902280549:84 , srv913852575:84 , srv913871437:84 , srv951075060:84 , srv955707499:84 , srv1434523497:85 , srv1695529014:85 , srv1953121625:85 , srv2081255497:85 , srv2143269026:85 , srv729785202:85 , srv1012002803:684 } 2023-07-24 12:14:51,913 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=307 (was 299) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=2196 (was 3064) 2023-07-24 12:14:51,925 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=265, MaxFileDescriptor=60000, SystemLoadAverage=307, ProcessCount=169, AvailableMemoryMB=2195 2023-07-24 12:14:51,926 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-24 12:14:51,927 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-07-24 12:14:51,940 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1011933999:84 , srv1014016713:84 , srv1045025839:84 , srv111122116:84 , srv117772216:84 , srv1268465623:84 , srv1377989121:84 , srv1501167556:84 , srv1513101209:84 , srv1549821961:84 , srv1586237510:84 , srv1626796772:84 , srv1750215167:84 , srv1803076886:84 , srv1877056990:84 , srv246300992:84 , srv402378254:84 , srv50707363:84 , srv51781968:84 , srv52364083:84 , srv611675475:84 , srv673281349:84 , srv728235321:84 , srv742479884:84 , srv767279560:84 , srv779473253:84 , srv844321114:84 , srv917635059:84 , srv991520884:84 , srv1777941049:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-07-24 12:14:51,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-24 12:14:51,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv728235321=23, srv673281349=22, srv52364083=20, srv1268465623=5, srv111122116=3, srv1014016713=1, srv1045025839=2, srv1011933999=0, srv1777941049=13, srv844321114=27, srv1501167556=7, srv50707363=18, srv1626796772=11, srv917635059=28, srv1750215167=12, srv1803076886=14, srv402378254=17, srv767279560=25, srv611675475=21, srv991520884=29, srv117772216=4, srv51781968=19, srv1513101209=8, srv1877056990=15, srv246300992=16, srv1377989121=6, srv779473253=26, srv742479884=24, srv1549821961=9, srv1586237510=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-24 12:14:51,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-24 12:14:51,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-24 12:14:51,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-24 12:14:51,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-24 12:14:51,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-24 12:14:51,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-24 12:14:51,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-24 12:14:51,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-24 12:14:51,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-24 12:14:51,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-24 12:14:51,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-24 12:14:51,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-24 12:14:51,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-24 12:14:51,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-24 12:14:51,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-24 12:14:51,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-24 12:14:51,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-24 12:14:51,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-24 12:14:51,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-24 12:14:51,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-24 12:14:51,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-24 12:14:51,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-24 12:14:51,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-24 12:14:51,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-24 12:14:51,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-24 12:14:51,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-24 12:14:51,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-24 12:14:51,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-24 12:14:51,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-24 12:14:51,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-24 12:14:51,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-07-24 12:14:51,976 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1669817960619084, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-24 12:14:51,976 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912719197451212, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1669817960619084, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-07-24 12:16:51,958 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6153212 different iterations. Found a solution that moves 1859 regions; Going from a computed imbalance of 0.9912719197451212 to a new imbalance of 4.338998739279625E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6885185185185185, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-24 12:16:51,967 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1011933999:90 , srv1014016713:90 , srv1045025839:90 , srv111122116:90 , srv117772216:90 , srv1268465623:90 , srv1377989121:90 , srv1501167556:90 , srv1513101209:90 , srv1549821961:90 , srv1586237510:90 , srv1626796772:90 , srv1750215167:90 , srv1777941049:90 , srv1803076886:90 , srv1877056990:90 , srv246300992:90 , srv402378254:90 , srv50707363:90 , srv51781968:90 , srv52364083:90 , srv611675475:90 , srv673281349:90 , srv728235321:90 , srv742479884:90 , srv767279560:90 , srv779473253:90 , srv844321114:90 , srv917635059:90 , srv991520884:90 } 2023-07-24 12:16:51,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-24 12:16:51,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv728235321=23, srv673281349=22, srv52364083=20, srv1268465623=5, srv111122116=3, srv1014016713=1, srv1045025839=2, srv1011933999=0, srv1777941049=13, srv844321114=27, srv1501167556=7, srv50707363=18, srv1626796772=11, srv917635059=28, srv1750215167=12, srv1803076886=14, srv402378254=17, srv767279560=25, srv611675475=21, srv991520884=29, srv117772216=4, srv51781968=19, srv1513101209=8, srv1877056990=15, srv246300992=16, srv1377989121=6, srv779473253=26, srv742479884=24, srv1549821961=9, srv1586237510=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-24 12:16:51,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-24 12:16:51,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-24 12:16:51,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-24 12:16:51,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-24 12:16:51,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-24 12:16:51,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-24 12:16:51,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-24 12:16:51,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-24 12:16:51,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-24 12:16:51,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-24 12:16:51,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-24 12:16:51,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-24 12:16:51,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-24 12:16:51,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-24 12:16:51,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-24 12:16:51,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-24 12:16:51,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-24 12:16:51,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-24 12:16:51,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-24 12:16:51,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-24 12:16:51,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-24 12:16:51,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-24 12:16:51,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-24 12:16:51,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-24 12:16:51,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-24 12:16:51,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-24 12:16:51,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-24 12:16:51,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-24 12:16:51,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-24 12:16:51,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-24 12:16:51,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-07-24 12:16:51,991 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-24 12:16:51,991 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1011933999:90 , srv1014016713:90 , srv1045025839:90 , srv111122116:90 , srv117772216:90 , srv1268465623:90 , srv1377989121:90 , srv1501167556:90 , srv1513101209:90 , srv1549821961:90 , srv1586237510:90 , srv1626796772:90 , srv1750215167:90 , srv1777941049:90 , srv1803076886:90 , srv1877056990:90 , srv246300992:90 , srv402378254:90 , srv50707363:90 , srv51781968:90 , srv52364083:90 , srv611675475:90 , srv673281349:90 , srv728235321:90 , srv742479884:90 , srv767279560:90 , srv779473253:90 , srv844321114:90 , srv917635059:90 , srv991520884:90 } 2023-07-24 12:16:52,006 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 265), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=189 (was 307), ProcessCount=168 (was 169), AvailableMemoryMB=2127 (was 2195)