2023-09-02 21:53:10,909 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-09-02 21:53:11,229 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-09-02 21:53:11,346 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-09-02 21:53:11,413 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-02 21:53:11,421 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-09-02 21:53:11,438 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=260, ProcessCount=170, AvailableMemoryMB=10229 2023-09-02 21:53:11,440 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-02 21:53:11,441 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-09-02 21:53:11,593 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1013370884:84 , srv1028349372:84 , srv1069415588:84 , srv1086893:84 , srv1091654784:84 , srv1092637401:84 , srv109429717:84 , srv1143948384:84 , srv1169574681:84 , srv118315496:84 , srv1194920539:84 , srv1204071651:84 , srv1206298455:84 , srv1230937422:84 , srv1245668846:84 , srv1251108228:84 , srv1253063985:84 , srv1270003400:84 , srv1281223963:84 , srv1299680888:84 , srv134280956:84 , srv1354814783:84 , srv1357789991:84 , srv1395357056:84 , srv1451685570:84 , srv1461530027:84 , srv1492321286:84 , srv1519939473:84 , srv1521904967:84 , srv1537294712:84 , srv1544769605:84 , srv1569628470:84 , srv1584562495:84 , srv1588609269:84 , srv1611359827:84 , srv1612461494:84 , srv1673199270:84 , srv1673690507:84 , srv1760415471:84 , srv177351103:84 , srv1803225295:84 , srv1806005528:84 , srv1893183280:84 , srv1914394972:84 , srv1935049574:84 , srv1937518356:84 , srv1971737756:84 , srv1976619397:84 , srv1989622864:84 , srv2001835590:84 , srv2033607295:84 , srv2034133105:84 , srv2071467131:84 , srv2072676619:84 , srv208975759:84 , srv2117697026:84 , srv2135643159:84 , srv2145773234:84 , srv216038143:84 , srv250652509:84 , srv278972396:84 , srv280016225:84 , srv31288772:84 , srv319809151:84 , srv324112223:84 , srv335536322:84 , srv370591131:84 , srv393765126:84 , srv417708809:84 , srv42304886:84 , srv443659970:84 , srv502989917:84 , srv505077222:84 , srv505820265:84 , srv514893409:84 , srv594470509:84 , srv596506819:84 , srv608832926:84 , srv673922733:84 , srv762042462:84 , srv77322967:84 , srv791160648:84 , srv794873426:84 , srv827037520:84 , srv833879167:84 , srv842246857:84 , srv845666338:84 , srv850203313:84 , srv875159682:84 , srv890327802:84 , srv902509414:84 , srv904339399:84 , srv909798646:84 , srv910930443:84 , srv917979906:84 , srv945772829:84 , srv956521099:84 , srv961900521:84 , srv99445292:84 , srv1916916653:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-09-02 21:53:11,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-02 21:53:11,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1612461494=35, srv370591131=67, srv1013370884=0, srv1893183280=42, srv909798646=93, srv1914394972=43, srv1937518356=46, srv177351103=39, srv1143948384=7, srv109429717=6, srv2072676619=54, srv1537294712=29, srv1544769605=30, srv794873426=83, srv1253063985=16, srv1760415471=38, srv1806005528=41, srv99445292=99, srv1354814783=21, srv42304886=70, srv1492321286=26, srv2145773234=58, srv1357789991=22, srv902509414=91, srv1935049574=45, srv1569628470=31, srv842246857=86, srv1245668846=14, srv1281223963=18, srv324112223=65, srv1673690507=37, srv833879167=85, srv319809151=64, srv77322967=81, srv335536322=66, srv1916916653=44, srv1521904967=28, srv31288772=63, srv875159682=89, srv1976619397=48, srv216038143=59, srv1206298455=12, srv1451685570=24, srv1270003400=17, srv280016225=62, srv505820265=74, srv1803225295=40, srv850203313=88, srv961900521=98, srv1584562495=32, srv1194920539=10, srv827037520=84, srv278972396=61, srv1461530027=25, srv1251108228=15, srv596506819=77, srv250652509=60, srv845666338=87, srv1086893=3, srv608832926=78, srv1611359827=34, srv594470509=76, srv2001835590=50, srv2034133105=52, srv417708809=69, srv1169574681=8, srv956521099=97, srv1588609269=33, srv673922733=79, srv1204071651=11, srv910930443=94, srv118315496=9, srv917979906=95, srv443659970=71, srv2071467131=53, srv1299680888=19, srv502989917=72, srv945772829=96, srv1091654784=4, srv1989622864=49, srv208975759=55, srv505077222=73, srv762042462=80, srv134280956=20, srv890327802=90, srv1092637401=5, srv904339399=92, srv1028349372=1, srv2033607295=51, srv1673199270=36, srv2135643159=57, srv1230937422=13, srv1519939473=27, srv791160648=82, srv1971737756=47, srv1069415588=2, srv1395357056=23, srv2117697026=56, srv393765126=68, srv514893409=75} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-02 21:53:11,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-02 21:53:11,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-02 21:53:11,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-02 21:53:11,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-02 21:53:11,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-02 21:53:11,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-02 21:53:11,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-02 21:53:11,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-02 21:53:11,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-02 21:53:11,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-02 21:53:11,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-02 21:53:11,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-02 21:53:11,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-02 21:53:11,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-02 21:53:11,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-02 21:53:11,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-02 21:53:11,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-02 21:53:11,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-02 21:53:11,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-02 21:53:11,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-02 21:53:11,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-02 21:53:11,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-02 21:53:11,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-02 21:53:11,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-02 21:53:11,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-02 21:53:11,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-02 21:53:11,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-02 21:53:11,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-02 21:53:11,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-02 21:53:11,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-02 21:53:11,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-02 21:53:11,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-02 21:53:11,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-02 21:53:11,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-02 21:53:11,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-02 21:53:11,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-02 21:53:11,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-02 21:53:11,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-02 21:53:11,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-02 21:53:11,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-02 21:53:11,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-02 21:53:11,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-02 21:53:11,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-02 21:53:11,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-02 21:53:11,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-02 21:53:11,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-02 21:53:11,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-02 21:53:11,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-02 21:53:11,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-02 21:53:11,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-02 21:53:11,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-02 21:53:11,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-02 21:53:11,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-02 21:53:11,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-02 21:53:11,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-02 21:53:11,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-02 21:53:11,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-02 21:53:11,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-02 21:53:11,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-02 21:53:11,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-02 21:53:11,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-02 21:53:11,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-02 21:53:11,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-02 21:53:11,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-02 21:53:11,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-02 21:53:11,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-02 21:53:11,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-02 21:53:11,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-02 21:53:11,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-02 21:53:11,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-02 21:53:11,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-02 21:53:11,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-02 21:53:11,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-02 21:53:11,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-02 21:53:11,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-02 21:53:11,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-02 21:53:11,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-02 21:53:11,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-02 21:53:11,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-02 21:53:11,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-02 21:53:11,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-02 21:53:11,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-02 21:53:11,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-02 21:53:11,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-02 21:53:11,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-02 21:53:11,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-02 21:53:11,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-02 21:53:11,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-02 21:53:11,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-02 21:53:11,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-02 21:53:11,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-02 21:53:11,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-02 21:53:11,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-02 21:53:11,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-02 21:53:11,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-02 21:53:11,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-02 21:53:11,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-02 21:53:11,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-02 21:53:11,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-02 21:53:11,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-02 21:53:11,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-02 21:53:12,172 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-02 21:53:12,172 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-02 21:53:12,173 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-02 21:53:12,893 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1168 ms to try 5000 different iterations. Found a solution that moves 4689 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18134746140556687. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0415172404792827, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05867982711280578, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.521, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0415172404792827, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1765, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24383333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-02 21:53:12,913 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1611359827:82 , srv1673690507:82 , srv177351103:82 , srv1028349372:83 , srv1245668846:83 , srv1251108228:83 , srv1354814783:83 , srv1461530027:83 , srv1569628470:83 , srv1588609269:83 , srv2135643159:83 , srv833879167:83 , srv99445292:83 , srv118315496:84 , srv1281223963:84 , srv1299680888:84 , srv1451685570:84 , srv1544769605:84 , srv1612461494:84 , srv1971737756:84 , srv2001835590:84 , srv216038143:84 , srv42304886:84 , srv505077222:84 , srv505820265:84 , srv594470509:84 , srv673922733:84 , srv850203313:84 , srv1143948384:85 , srv1194920539:85 , srv134280956:85 , srv1357789991:85 , srv1519939473:85 , srv1673199270:85 , srv1806005528:85 , srv1893183280:85 , srv1914394972:85 , srv2117697026:85 , srv608832926:85 , srv762042462:85 , srv875159682:85 , srv890327802:85 , srv904339399:85 , srv910930443:85 , srv961900521:85 , srv1013370884:86 , srv1069415588:86 , srv1169574681:86 , srv1270003400:86 , srv1492321286:86 , srv1537294712:86 , srv1760415471:86 , srv1803225295:86 , srv514893409:86 , srv77322967:86 , srv791160648:86 , srv794873426:86 , srv845666338:86 , srv917979906:86 , srv1092637401:87 , srv1204071651:87 , srv1230937422:87 , srv1521904967:87 , srv1976619397:87 , srv2033607295:87 , srv278972396:87 , srv443659970:87 , srv902509414:87 , srv909798646:87 , srv956521099:87 , srv109429717:88 , srv1253063985:88 , srv1937518356:88 , srv2072676619:88 , srv208975759:88 , srv2145773234:88 , srv31288772:88 , srv335536322:88 , srv393765126:88 , srv502989917:88 , srv945772829:88 , srv1206298455:89 , srv2034133105:89 , srv2071467131:89 , srv370591131:89 , srv842246857:89 , srv1091654784:90 , srv1935049574:90 , srv324112223:90 , srv417708809:90 , srv596506819:90 , srv1584562495:91 , srv1989622864:91 , srv319809151:91 , srv827037520:91 , srv250652509:92 , srv1086893:93 , srv280016225:93 , srv1395357056:94 , srv1916916653:459 } 2023-09-02 21:53:12,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-02 21:53:12,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1612461494=35, srv370591131=67, srv1013370884=0, srv1893183280=42, srv909798646=93, srv1914394972=43, srv1937518356=46, srv177351103=39, srv1143948384=7, srv109429717=6, srv2072676619=54, srv1537294712=29, srv1544769605=30, srv794873426=83, srv1253063985=16, srv1760415471=38, srv1806005528=41, srv99445292=99, srv1354814783=21, srv42304886=70, srv1492321286=26, srv2145773234=58, srv1357789991=22, srv902509414=91, srv1935049574=45, srv1569628470=31, srv842246857=86, srv1245668846=14, srv1281223963=18, srv324112223=65, srv1673690507=37, srv833879167=85, srv319809151=64, srv77322967=81, srv335536322=66, srv1916916653=44, srv1521904967=28, srv31288772=63, srv875159682=89, srv1976619397=48, srv216038143=59, srv1206298455=12, srv1451685570=24, srv1270003400=17, srv280016225=62, srv505820265=74, srv1803225295=40, srv850203313=88, srv961900521=98, srv1584562495=32, srv1194920539=10, srv827037520=84, srv278972396=61, srv1461530027=25, srv1251108228=15, srv596506819=77, srv250652509=60, srv845666338=87, srv1086893=3, srv608832926=78, srv1611359827=34, srv594470509=76, srv2001835590=50, srv2034133105=52, srv417708809=69, srv1169574681=8, srv956521099=97, srv1588609269=33, srv673922733=79, srv1204071651=11, srv910930443=94, srv118315496=9, srv917979906=95, srv443659970=71, srv2071467131=53, srv1299680888=19, srv502989917=72, srv945772829=96, srv1091654784=4, srv1989622864=49, srv208975759=55, srv505077222=73, srv762042462=80, srv134280956=20, srv890327802=90, srv1092637401=5, srv904339399=92, srv1028349372=1, srv2033607295=51, srv1673199270=36, srv2135643159=57, srv1230937422=13, srv1519939473=27, srv791160648=82, srv1971737756=47, srv1069415588=2, srv1395357056=23, srv2117697026=56, srv393765126=68, srv514893409=75} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-02 21:53:12,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-02 21:53:12,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-02 21:53:12,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-02 21:53:12,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-02 21:53:12,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-02 21:53:12,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-02 21:53:12,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-02 21:53:12,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-02 21:53:12,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-02 21:53:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-02 21:53:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-02 21:53:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-02 21:53:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-02 21:53:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-02 21:53:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-02 21:53:12,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-02 21:53:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-02 21:53:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-02 21:53:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-02 21:53:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-02 21:53:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-02 21:53:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-02 21:53:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-02 21:53:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-02 21:53:12,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-02 21:53:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-02 21:53:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-02 21:53:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-02 21:53:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-02 21:53:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-02 21:53:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-02 21:53:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-02 21:53:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-02 21:53:12,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-02 21:53:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-02 21:53:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-02 21:53:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-02 21:53:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-02 21:53:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-02 21:53:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-02 21:53:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-02 21:53:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-02 21:53:12,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-02 21:53:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-02 21:53:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-02 21:53:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-02 21:53:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-02 21:53:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-02 21:53:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-02 21:53:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-02 21:53:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-02 21:53:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-02 21:53:12,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-02 21:53:12,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-02 21:53:12,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-02 21:53:12,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-02 21:53:12,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-02 21:53:12,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-02 21:53:12,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-02 21:53:12,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-02 21:53:12,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-02 21:53:12,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-02 21:53:12,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-02 21:53:12,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-02 21:53:12,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-02 21:53:12,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-02 21:53:12,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-02 21:53:12,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-02 21:53:12,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-02 21:53:12,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-02 21:53:12,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-02 21:53:12,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-02 21:53:12,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-02 21:53:12,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-02 21:53:12,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-02 21:53:12,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-02 21:53:12,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-02 21:53:12,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-02 21:53:12,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-02 21:53:12,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-02 21:53:12,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-02 21:53:12,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-02 21:53:12,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-02 21:53:12,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-02 21:53:12,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-02 21:53:12,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-02 21:53:12,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-02 21:53:12,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-02 21:53:12,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-02 21:53:12,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-02 21:53:12,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-02 21:53:12,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-02 21:53:12,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-02 21:53:12,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-02 21:53:12,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-02 21:53:12,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-02 21:53:12,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-02 21:53:12,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-02 21:53:12,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-02 21:53:12,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-02 21:53:12,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-02 21:53:13,253 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0415172404792827, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05867982711280578, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0415172404792827, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1765, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24383333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-02 21:53:13,253 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-02 21:53:13,253 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18131462832581138, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0415172404792827, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05867982711280578, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0415172404792827, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1765, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24383333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-02 21:53:13,577 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 586 ms to try 5000 different iterations. Found a solution that moves 2620 regions; Going from a computed imbalance of 0.18131462832581138 to a new imbalance of 0.015693045669946924. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02329134161740036); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03894224334188153, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2911111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02329134161740036); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-02 21:53:13,588 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1395357056:76 , srv1086893:77 , srv280016225:79 , srv319809151:80 , srv324112223:80 , srv827037520:80 , srv1989622864:81 , srv1091654784:82 , srv1253063985:82 , srv1584562495:82 , srv2071467131:82 , srv250652509:82 , srv278972396:82 , srv393765126:82 , srv956521099:82 , srv109429717:83 , srv1206298455:83 , srv2034133105:83 , srv417708809:83 , srv514893409:83 , srv842246857:83 , srv1204071651:84 , srv1803225295:84 , srv1935049574:84 , srv1976619397:84 , srv208975759:84 , srv502989917:84 , srv596506819:84 , srv845666338:84 , srv902509414:84 , srv1069415588:85 , srv1092637401:85 , srv1451685570:85 , srv1914394972:85 , srv1937518356:85 , srv31288772:85 , srv335536322:85 , srv443659970:85 , srv77322967:85 , srv917979906:85 , srv1013370884:86 , srv1143948384:86 , srv1169574681:86 , srv1492321286:86 , srv1521904967:86 , srv1537294712:86 , srv1673199270:86 , srv2033607295:86 , srv2072676619:86 , srv2145773234:86 , srv370591131:86 , srv762042462:86 , srv794873426:86 , srv909798646:86 , srv910930443:86 , srv945772829:86 , srv118315496:87 , srv1194920539:87 , srv1270003400:87 , srv134280956:87 , srv1519939473:87 , srv1544769605:87 , srv1612461494:87 , srv1893183280:87 , srv1971737756:87 , srv2001835590:87 , srv2135643159:87 , srv505820265:87 , srv833879167:87 , srv850203313:87 , srv99445292:87 , srv1245668846:88 , srv1251108228:88 , srv1281223963:88 , srv1299680888:88 , srv1354814783:88 , srv1673690507:88 , srv1760415471:88 , srv1806005528:88 , srv2117697026:88 , srv42304886:88 , srv505077222:88 , srv673922733:88 , srv791160648:88 , srv875159682:88 , srv890327802:88 , srv904339399:88 , srv1230937422:89 , srv1357789991:89 , srv1588609269:89 , srv177351103:89 , srv216038143:89 , srv961900521:89 , srv1028349372:90 , srv1461530027:90 , srv1569628470:90 , srv1611359827:90 , srv594470509:90 , srv608832926:92 , srv1916916653:522 } 2023-09-02 21:53:13,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-02 21:53:13,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1612461494=35, srv370591131=67, srv1013370884=0, srv1893183280=42, srv909798646=93, srv1914394972=43, srv1937518356=46, srv177351103=39, srv1143948384=7, srv109429717=6, srv2072676619=54, srv1537294712=29, srv1544769605=30, srv794873426=83, srv1253063985=16, srv1760415471=38, srv1806005528=41, srv99445292=99, srv1354814783=21, srv42304886=70, srv1492321286=26, srv2145773234=58, srv1357789991=22, srv902509414=91, srv1935049574=45, srv1569628470=31, srv842246857=86, srv1245668846=14, srv1281223963=18, srv324112223=65, srv1673690507=37, srv833879167=85, srv319809151=64, srv77322967=81, srv335536322=66, srv1916916653=44, srv1521904967=28, srv31288772=63, srv875159682=89, srv1976619397=48, srv216038143=59, srv1206298455=12, srv1451685570=24, srv1270003400=17, srv280016225=62, srv505820265=74, srv1803225295=40, srv850203313=88, srv961900521=98, srv1584562495=32, srv1194920539=10, srv827037520=84, srv278972396=61, srv1461530027=25, srv1251108228=15, srv596506819=77, srv250652509=60, srv845666338=87, srv1086893=3, srv608832926=78, srv1611359827=34, srv594470509=76, srv2001835590=50, srv2034133105=52, srv417708809=69, srv1169574681=8, srv956521099=97, srv1588609269=33, srv673922733=79, srv1204071651=11, srv910930443=94, srv118315496=9, srv917979906=95, srv443659970=71, srv2071467131=53, srv1299680888=19, srv502989917=72, srv945772829=96, srv1091654784=4, srv1989622864=49, srv208975759=55, srv505077222=73, srv762042462=80, srv134280956=20, srv890327802=90, srv1092637401=5, srv904339399=92, srv1028349372=1, srv2033607295=51, srv1673199270=36, srv2135643159=57, srv1230937422=13, srv1519939473=27, srv791160648=82, srv1971737756=47, srv1069415588=2, srv1395357056=23, srv2117697026=56, srv393765126=68, srv514893409=75} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-02 21:53:13,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-02 21:53:13,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-02 21:53:13,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-02 21:53:13,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-02 21:53:13,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-02 21:53:13,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-02 21:53:13,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-02 21:53:13,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-02 21:53:13,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-02 21:53:13,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-02 21:53:13,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-02 21:53:13,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-02 21:53:13,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-02 21:53:13,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-02 21:53:13,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-02 21:53:13,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-02 21:53:13,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-02 21:53:13,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-02 21:53:13,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-02 21:53:13,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-02 21:53:13,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-02 21:53:13,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-02 21:53:13,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-02 21:53:13,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-02 21:53:13,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-02 21:53:13,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-02 21:53:13,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-02 21:53:13,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-02 21:53:13,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-02 21:53:13,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-02 21:53:13,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-02 21:53:13,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-02 21:53:13,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-02 21:53:13,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-02 21:53:13,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-02 21:53:13,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-02 21:53:13,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-02 21:53:13,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-02 21:53:13,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-02 21:53:13,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-02 21:53:13,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-02 21:53:13,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-02 21:53:13,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-02 21:53:13,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-02 21:53:13,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-02 21:53:13,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-02 21:53:13,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-02 21:53:13,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-02 21:53:13,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-02 21:53:13,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-02 21:53:13,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-02 21:53:13,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-02 21:53:13,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-02 21:53:13,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-02 21:53:13,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-02 21:53:13,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-02 21:53:13,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-02 21:53:13,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-02 21:53:13,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-02 21:53:13,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-02 21:53:13,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-02 21:53:13,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-02 21:53:13,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-02 21:53:13,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-02 21:53:13,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-02 21:53:13,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-02 21:53:13,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-02 21:53:13,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-02 21:53:13,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-02 21:53:13,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-02 21:53:13,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-02 21:53:13,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-02 21:53:13,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-02 21:53:13,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-02 21:53:13,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-02 21:53:13,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-02 21:53:13,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-02 21:53:13,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-02 21:53:13,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-02 21:53:13,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-02 21:53:13,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-02 21:53:13,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-02 21:53:13,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-02 21:53:13,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-02 21:53:13,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-02 21:53:13,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-02 21:53:13,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-02 21:53:13,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-02 21:53:13,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-02 21:53:13,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-02 21:53:13,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-02 21:53:13,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-02 21:53:13,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-02 21:53:13,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-02 21:53:13,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-02 21:53:13,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-02 21:53:13,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-02 21:53:13,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-02 21:53:13,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-02 21:53:13,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-02 21:53:13,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-02 21:53:13,787 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02329134161740036); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03894224334188153, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02329134161740036); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-02 21:53:13,787 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-02 21:53:13,788 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015674700037837868, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02329134161740036); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03894224334188153, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02329134161740036); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-02 21:53:14,064 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 399 ms to try 5000 different iterations. Found a solution that moves 1004 regions; Going from a computed imbalance of 0.015674700037837868 to a new imbalance of 4.882406245520715E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007466097461068055); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008914105645620285); MoveCostFunction : (multiplier=7.0, imbalance=0.11155555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007466097461068055); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0028333333333333335); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-02 21:53:14,066 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1230937422:80 , srv608832926:80 , srv370591131:81 , srv1357789991:82 , srv594470509:83 , srv791160648:83 , srv945772829:83 , srv961900521:83 , srv1028349372:84 , srv1206298455:84 , srv1461530027:84 , srv1521904967:84 , srv1584562495:84 , srv1760415471:84 , srv1806005528:84 , srv1935049574:84 , srv1937518356:84 , srv2034133105:84 , srv2072676619:84 , srv2117697026:84 , srv250652509:84 , srv417708809:84 , srv42304886:84 , srv596506819:84 , srv875159682:84 , srv909798646:84 , srv1013370884:85 , srv1169574681:85 , srv1194920539:85 , srv1251108228:85 , srv1281223963:85 , srv1492321286:85 , srv1519939473:85 , srv1537294712:85 , srv1569628470:85 , srv1588609269:85 , srv2033607295:85 , srv216038143:85 , srv31288772:85 , srv335536322:85 , srv443659970:85 , srv502989917:85 , srv505077222:85 , srv673922733:85 , srv794873426:85 , srv842246857:85 , srv902509414:85 , srv904339399:85 , srv1069415588:86 , srv1091654784:86 , srv1092637401:86 , srv118315496:86 , srv1245668846:86 , srv1270003400:86 , srv1299680888:86 , srv1354814783:86 , srv1544769605:86 , srv1611359827:86 , srv1612461494:86 , srv1673690507:86 , srv177351103:86 , srv1914394972:86 , srv1971737756:86 , srv1976619397:86 , srv1989622864:86 , srv2071467131:86 , srv208975759:86 , srv2145773234:86 , srv280016225:86 , srv319809151:86 , srv393765126:86 , srv762042462:86 , srv827037520:86 , srv850203313:86 , srv890327802:86 , srv910930443:86 , srv917979906:86 , srv1086893:87 , srv109429717:87 , srv1143948384:87 , srv1204071651:87 , srv1253063985:87 , srv1395357056:87 , srv1673199270:87 , srv1803225295:87 , srv1893183280:87 , srv2001835590:87 , srv324112223:87 , srv505820265:87 , srv77322967:87 , srv833879167:87 , srv1451685570:88 , srv2135643159:88 , srv278972396:88 , srv845666338:88 , srv134280956:89 , srv514893409:89 , srv956521099:89 , srv99445292:89 , srv1916916653:543 } 2023-09-02 21:53:14,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-02 21:53:14,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1612461494=35, srv370591131=67, srv1013370884=0, srv1893183280=42, srv909798646=93, srv1914394972=43, srv1937518356=46, srv177351103=39, srv1143948384=7, srv109429717=6, srv2072676619=54, srv1537294712=29, srv1544769605=30, srv794873426=83, srv1253063985=16, srv1760415471=38, srv1806005528=41, srv99445292=99, srv1354814783=21, srv42304886=70, srv1492321286=26, srv2145773234=58, srv1357789991=22, srv902509414=91, srv1935049574=45, srv1569628470=31, srv842246857=86, srv1245668846=14, srv1281223963=18, srv324112223=65, srv1673690507=37, srv833879167=85, srv319809151=64, srv77322967=81, srv335536322=66, srv1916916653=44, srv1521904967=28, srv31288772=63, srv875159682=89, srv1976619397=48, srv216038143=59, srv1206298455=12, srv1451685570=24, srv1270003400=17, srv280016225=62, srv505820265=74, srv1803225295=40, srv850203313=88, srv961900521=98, srv1584562495=32, srv1194920539=10, srv827037520=84, srv278972396=61, srv1461530027=25, srv1251108228=15, srv596506819=77, srv250652509=60, srv845666338=87, srv1086893=3, srv608832926=78, srv1611359827=34, srv594470509=76, srv2001835590=50, srv2034133105=52, srv417708809=69, srv1169574681=8, srv956521099=97, srv1588609269=33, srv673922733=79, srv1204071651=11, srv910930443=94, srv118315496=9, srv917979906=95, srv443659970=71, srv2071467131=53, srv1299680888=19, srv502989917=72, srv945772829=96, srv1091654784=4, srv1989622864=49, srv208975759=55, srv505077222=73, srv762042462=80, srv134280956=20, srv890327802=90, srv1092637401=5, srv904339399=92, srv1028349372=1, srv2033607295=51, srv1673199270=36, srv2135643159=57, srv1230937422=13, srv1519939473=27, srv791160648=82, srv1971737756=47, srv1069415588=2, srv1395357056=23, srv2117697026=56, srv393765126=68, srv514893409=75} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-02 21:53:14,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-02 21:53:14,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-02 21:53:14,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-02 21:53:14,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-02 21:53:14,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-02 21:53:14,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-02 21:53:14,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-02 21:53:14,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-02 21:53:14,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-02 21:53:14,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-02 21:53:14,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-02 21:53:14,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-02 21:53:14,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-02 21:53:14,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-02 21:53:14,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-02 21:53:14,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-02 21:53:14,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-02 21:53:14,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-02 21:53:14,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-02 21:53:14,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-02 21:53:14,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-02 21:53:14,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-02 21:53:14,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-02 21:53:14,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-02 21:53:14,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-02 21:53:14,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-02 21:53:14,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-02 21:53:14,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-02 21:53:14,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-02 21:53:14,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-02 21:53:14,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-02 21:53:14,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-02 21:53:14,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-02 21:53:14,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-02 21:53:14,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-02 21:53:14,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-02 21:53:14,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-02 21:53:14,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-02 21:53:14,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-02 21:53:14,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-02 21:53:14,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-02 21:53:14,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-02 21:53:14,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-02 21:53:14,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-02 21:53:14,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-02 21:53:14,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-02 21:53:14,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-02 21:53:14,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-02 21:53:14,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-02 21:53:14,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-02 21:53:14,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-02 21:53:14,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-02 21:53:14,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-02 21:53:14,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-02 21:53:14,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-02 21:53:14,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-02 21:53:14,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-02 21:53:14,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-02 21:53:14,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-02 21:53:14,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-02 21:53:14,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-02 21:53:14,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-02 21:53:14,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-02 21:53:14,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-02 21:53:14,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-02 21:53:14,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-02 21:53:14,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-02 21:53:14,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-02 21:53:14,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-02 21:53:14,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-02 21:53:14,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-02 21:53:14,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-02 21:53:14,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-02 21:53:14,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-02 21:53:14,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-02 21:53:14,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-02 21:53:14,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-02 21:53:14,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-02 21:53:14,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-02 21:53:14,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-02 21:53:14,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-02 21:53:14,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-02 21:53:14,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-02 21:53:14,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-02 21:53:14,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-02 21:53:14,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-02 21:53:14,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-02 21:53:14,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-02 21:53:14,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-02 21:53:14,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-02 21:53:14,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-02 21:53:14,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-02 21:53:14,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-02 21:53:14,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-02 21:53:14,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-02 21:53:14,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-02 21:53:14,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-02 21:53:14,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-02 21:53:14,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-02 21:53:14,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-02 21:53:14,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-02 21:53:14,225 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007466097461068055); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008914105645620285); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007466097461068055); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0028333333333333335); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-02 21:53:14,225 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-09-02 21:53:14,226 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.812104662934861E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007466097461068055); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008914105645620285); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007466097461068055); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0028333333333333335); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-09-02 21:53:14,607 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 503 ms to try 5000 different iterations. Found a solution that moves 224 regions; Going from a computed imbalance of 4.812104662934861E-4 to a new imbalance of 1.3015980603545853E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.024888888888888887); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-02 21:53:14,608 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv134280956:81 , srv1270003400:83 , srv1673199270:83 , srv1893183280:83 , srv2145773234:83 , srv890327802:83 , srv99445292:83 , srv1091654784:84 , srv1092637401:84 , srv109429717:84 , srv1143948384:84 , srv1204071651:84 , srv1299680888:84 , srv1451685570:84 , srv1569628470:84 , srv1584562495:84 , srv1611359827:84 , srv1935049574:84 , srv1989622864:84 , srv2001835590:84 , srv2033607295:84 , srv2071467131:84 , srv2072676619:84 , srv208975759:84 , srv2135643159:84 , srv216038143:84 , srv250652509:84 , srv280016225:84 , srv31288772:84 , srv335536322:84 , srv502989917:84 , srv514893409:84 , srv596506819:84 , srv762042462:84 , srv77322967:84 , srv845666338:84 , srv904339399:84 , srv909798646:84 , srv956521099:84 , srv1013370884:85 , srv1028349372:85 , srv1069415588:85 , srv1086893:85 , srv1169574681:85 , srv118315496:85 , srv1194920539:85 , srv1245668846:85 , srv1253063985:85 , srv1281223963:85 , srv1354814783:85 , srv1461530027:85 , srv1492321286:85 , srv1519939473:85 , srv1521904967:85 , srv1537294712:85 , srv1588609269:85 , srv1612461494:85 , srv1760415471:85 , srv177351103:85 , srv1803225295:85 , srv1806005528:85 , srv1937518356:85 , srv1971737756:85 , srv1976619397:85 , srv2117697026:85 , srv278972396:85 , srv319809151:85 , srv417708809:85 , srv443659970:85 , srv505077222:85 , srv505820265:85 , srv594470509:85 , srv608832926:85 , srv673922733:85 , srv791160648:85 , srv794873426:85 , srv827037520:85 , srv833879167:85 , srv842246857:85 , srv850203313:85 , srv875159682:85 , srv902509414:85 , srv917979906:85 , srv945772829:85 , srv961900521:85 , srv1206298455:86 , srv1251108228:86 , srv1357789991:86 , srv1395357056:86 , srv1544769605:86 , srv1673690507:86 , srv1914394972:86 , srv2034133105:86 , srv324112223:86 , srv370591131:86 , srv393765126:86 , srv42304886:86 , srv910930443:86 , srv1230937422:87 , srv1916916653:618 } 2023-09-02 21:53:14,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-02 21:53:14,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1612461494=35, srv370591131=67, srv1013370884=0, srv1893183280=42, srv909798646=93, srv1914394972=43, srv1937518356=46, srv177351103=39, srv1143948384=7, srv109429717=6, srv2072676619=54, srv1537294712=29, srv1544769605=30, srv794873426=83, srv1253063985=16, srv1760415471=38, srv1806005528=41, srv99445292=99, srv1354814783=21, srv42304886=70, srv1492321286=26, srv2145773234=58, srv1357789991=22, srv902509414=91, srv1935049574=45, srv1569628470=31, srv842246857=86, srv1245668846=14, srv1281223963=18, srv324112223=65, srv1673690507=37, srv833879167=85, srv319809151=64, srv77322967=81, srv335536322=66, srv1916916653=44, srv1521904967=28, srv31288772=63, srv875159682=89, srv1976619397=48, srv216038143=59, srv1206298455=12, srv1451685570=24, srv1270003400=17, srv280016225=62, srv505820265=74, srv1803225295=40, srv850203313=88, srv961900521=98, srv1584562495=32, srv1194920539=10, srv827037520=84, srv278972396=61, srv1461530027=25, srv1251108228=15, srv596506819=77, srv250652509=60, srv845666338=87, srv1086893=3, srv608832926=78, srv1611359827=34, srv594470509=76, srv2001835590=50, srv2034133105=52, srv417708809=69, srv1169574681=8, srv956521099=97, srv1588609269=33, srv673922733=79, srv1204071651=11, srv910930443=94, srv118315496=9, srv917979906=95, srv443659970=71, srv2071467131=53, srv1299680888=19, srv502989917=72, srv945772829=96, srv1091654784=4, srv1989622864=49, srv208975759=55, srv505077222=73, srv762042462=80, srv134280956=20, srv890327802=90, srv1092637401=5, srv904339399=92, srv1028349372=1, srv2033607295=51, srv1673199270=36, srv2135643159=57, srv1230937422=13, srv1519939473=27, srv791160648=82, srv1971737756=47, srv1069415588=2, srv1395357056=23, srv2117697026=56, srv393765126=68, srv514893409=75} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-02 21:53:14,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-02 21:53:14,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-02 21:53:14,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-02 21:53:14,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-02 21:53:14,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-02 21:53:14,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-02 21:53:14,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-02 21:53:14,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-02 21:53:14,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-02 21:53:14,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-02 21:53:14,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-02 21:53:14,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-02 21:53:14,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-02 21:53:14,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-02 21:53:14,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-02 21:53:14,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-02 21:53:14,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-02 21:53:14,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-02 21:53:14,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-02 21:53:14,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-02 21:53:14,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-02 21:53:14,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-02 21:53:14,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-02 21:53:14,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-02 21:53:14,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-02 21:53:14,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-02 21:53:14,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-02 21:53:14,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-02 21:53:14,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-02 21:53:14,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-02 21:53:14,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-09-02 21:53:14,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-09-02 21:53:14,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-09-02 21:53:14,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-09-02 21:53:14,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-09-02 21:53:14,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-09-02 21:53:14,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-09-02 21:53:14,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-09-02 21:53:14,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-09-02 21:53:14,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-09-02 21:53:14,629 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-09-02 21:53:14,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-09-02 21:53:14,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-09-02 21:53:14,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-09-02 21:53:14,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-09-02 21:53:14,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-09-02 21:53:14,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-09-02 21:53:14,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-09-02 21:53:14,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-09-02 21:53:14,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-09-02 21:53:14,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-09-02 21:53:14,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-09-02 21:53:14,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-09-02 21:53:14,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-09-02 21:53:14,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-09-02 21:53:14,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-09-02 21:53:14,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-09-02 21:53:14,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-09-02 21:53:14,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-09-02 21:53:14,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-09-02 21:53:14,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-09-02 21:53:14,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-09-02 21:53:14,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-09-02 21:53:14,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-09-02 21:53:14,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-09-02 21:53:14,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-09-02 21:53:14,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-09-02 21:53:14,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-09-02 21:53:14,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-09-02 21:53:14,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-09-02 21:53:14,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-09-02 21:53:14,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-09-02 21:53:14,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-09-02 21:53:14,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-09-02 21:53:14,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-09-02 21:53:14,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-09-02 21:53:14,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-09-02 21:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-09-02 21:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-09-02 21:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-09-02 21:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-09-02 21:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-09-02 21:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-09-02 21:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-09-02 21:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-09-02 21:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-09-02 21:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-09-02 21:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-09-02 21:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-09-02 21:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-09-02 21:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-09-02 21:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-09-02 21:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-09-02 21:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-09-02 21:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-09-02 21:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-09-02 21:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-09-02 21:53:14,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-09-02 21:53:14,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-09-02 21:53:14,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-09-02 21:53:14,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-09-02 21:53:14,751 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.1447499079718037E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-02 21:53:14,752 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv134280956:81 , srv1270003400:83 , srv1673199270:83 , srv1893183280:83 , srv2145773234:83 , srv890327802:83 , srv99445292:83 , srv1091654784:84 , srv1092637401:84 , srv109429717:84 , srv1143948384:84 , srv1204071651:84 , srv1299680888:84 , srv1451685570:84 , srv1569628470:84 , srv1584562495:84 , srv1611359827:84 , srv1935049574:84 , srv1989622864:84 , srv2001835590:84 , srv2033607295:84 , srv2071467131:84 , srv2072676619:84 , srv208975759:84 , srv2135643159:84 , srv216038143:84 , srv250652509:84 , srv280016225:84 , srv31288772:84 , srv335536322:84 , srv502989917:84 , srv514893409:84 , srv596506819:84 , srv762042462:84 , srv77322967:84 , srv845666338:84 , srv904339399:84 , srv909798646:84 , srv956521099:84 , srv1013370884:85 , srv1028349372:85 , srv1069415588:85 , srv1086893:85 , srv1169574681:85 , srv118315496:85 , srv1194920539:85 , srv1245668846:85 , srv1253063985:85 , srv1281223963:85 , srv1354814783:85 , srv1461530027:85 , srv1492321286:85 , srv1519939473:85 , srv1521904967:85 , srv1537294712:85 , srv1588609269:85 , srv1612461494:85 , srv1760415471:85 , srv177351103:85 , srv1803225295:85 , srv1806005528:85 , srv1937518356:85 , srv1971737756:85 , srv1976619397:85 , srv2117697026:85 , srv278972396:85 , srv319809151:85 , srv417708809:85 , srv443659970:85 , srv505077222:85 , srv505820265:85 , srv594470509:85 , srv608832926:85 , srv673922733:85 , srv791160648:85 , srv794873426:85 , srv827037520:85 , srv833879167:85 , srv842246857:85 , srv850203313:85 , srv875159682:85 , srv902509414:85 , srv917979906:85 , srv945772829:85 , srv961900521:85 , srv1206298455:86 , srv1251108228:86 , srv1357789991:86 , srv1395357056:86 , srv1544769605:86 , srv1673690507:86 , srv1914394972:86 , srv2034133105:86 , srv324112223:86 , srv370591131:86 , srv393765126:86 , srv42304886:86 , srv910930443:86 , srv1230937422:87 , srv1916916653:618 } 2023-09-02 21:53:14,800 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=279 (was 260) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=9358 (was 10229) 2023-09-02 21:53:14,814 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=279, ProcessCount=170, AvailableMemoryMB=9357 2023-09-02 21:53:14,815 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-09-02 21:53:14,815 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-09-02 21:53:14,831 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1019142671:84 , srv1050536873:84 , srv1128897916:84 , srv1221599058:84 , srv1369770028:84 , srv1449076568:84 , srv1473984481:84 , srv1489520588:84 , srv1608941952:84 , srv1614225351:84 , srv1711473201:84 , srv1789334003:84 , srv1930276033:84 , srv1957955768:84 , srv2032702341:84 , srv2076626413:84 , srv2084542689:84 , srv2121394197:84 , srv2122005178:84 , srv566294053:84 , srv69930587:84 , srv746155638:84 , srv792855642:84 , srv816211670:84 , srv877873678:84 , srv887644144:84 , srv906036428:84 , srv95373157:84 , srv992135118:84 , srv1840003123:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-09-02 21:53:14,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-02 21:53:14,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1449076568=5, srv1473984481=6, srv1050536873=1, srv1711473201=10, srv1128897916=2, srv1957955768=14, srv1019142671=0, srv1369770028=4, srv1840003123=12, srv2076626413=16, srv1614225351=9, srv887644144=26, srv2032702341=15, srv2121394197=18, srv2122005178=19, srv95373157=28, srv1489520588=7, srv1608941952=8, srv1789334003=11, srv2084542689=17, srv746155638=22, srv69930587=21, srv792855642=23, srv566294053=20, srv877873678=25, srv906036428=27, srv992135118=29, srv1221599058=3, srv1930276033=13, srv816211670=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-02 21:53:14,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-02 21:53:14,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-02 21:53:14,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-02 21:53:14,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-02 21:53:14,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-02 21:53:14,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-02 21:53:14,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-02 21:53:14,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-02 21:53:14,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-02 21:53:14,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-02 21:53:14,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-02 21:53:14,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-02 21:53:14,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-02 21:53:14,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-02 21:53:14,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-02 21:53:14,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-02 21:53:14,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-02 21:53:14,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-02 21:53:14,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-02 21:53:14,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-02 21:53:14,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-02 21:53:14,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-02 21:53:14,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-02 21:53:14,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-02 21:53:14,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-02 21:53:14,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-02 21:53:14,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-02 21:53:14,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-02 21:53:14,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-02 21:53:14,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-02 21:53:14,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-09-02 21:53:14,867 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1804899812842395, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-02 21:53:14,868 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912761761301765, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1804899812842395, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-09-02 21:55:14,851 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6340505 different iterations. Found a solution that moves 1867 regions; Going from a computed imbalance of 0.9912761761301765 to a new imbalance of 4.3576711383728133E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6914814814814815, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-02 21:55:14,858 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1019142671:90 , srv1050536873:90 , srv1128897916:90 , srv1221599058:90 , srv1369770028:90 , srv1449076568:90 , srv1473984481:90 , srv1489520588:90 , srv1608941952:90 , srv1614225351:90 , srv1711473201:90 , srv1789334003:90 , srv1840003123:90 , srv1930276033:90 , srv1957955768:90 , srv2032702341:90 , srv2076626413:90 , srv2084542689:90 , srv2121394197:90 , srv2122005178:90 , srv566294053:90 , srv69930587:90 , srv746155638:90 , srv792855642:90 , srv816211670:90 , srv877873678:90 , srv887644144:90 , srv906036428:90 , srv95373157:90 , srv992135118:90 } 2023-09-02 21:55:14,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-09-02 21:55:14,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1449076568=5, srv1473984481=6, srv1050536873=1, srv1711473201=10, srv1128897916=2, srv1957955768=14, srv1019142671=0, srv1369770028=4, srv1840003123=12, srv2076626413=16, srv1614225351=9, srv887644144=26, srv2032702341=15, srv2121394197=18, srv2122005178=19, srv95373157=28, srv1489520588=7, srv1608941952=8, srv1789334003=11, srv2084542689=17, srv746155638=22, srv69930587=21, srv792855642=23, srv566294053=20, srv877873678=25, srv906036428=27, srv992135118=29, srv1221599058=3, srv1930276033=13, srv816211670=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-09-02 21:55:14,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-09-02 21:55:14,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-09-02 21:55:14,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-09-02 21:55:14,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-09-02 21:55:14,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-09-02 21:55:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-09-02 21:55:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-09-02 21:55:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-09-02 21:55:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-09-02 21:55:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-09-02 21:55:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-09-02 21:55:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-09-02 21:55:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-09-02 21:55:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-09-02 21:55:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-09-02 21:55:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-09-02 21:55:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-09-02 21:55:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-09-02 21:55:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-09-02 21:55:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-09-02 21:55:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-09-02 21:55:14,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-09-02 21:55:14,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-09-02 21:55:14,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-09-02 21:55:14,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-09-02 21:55:14,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-09-02 21:55:14,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-09-02 21:55:14,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-09-02 21:55:14,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-09-02 21:55:14,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-09-02 21:55:14,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-09-02 21:55:14,883 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-09-02 21:55:14,883 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1019142671:90 , srv1050536873:90 , srv1128897916:90 , srv1221599058:90 , srv1369770028:90 , srv1449076568:90 , srv1473984481:90 , srv1489520588:90 , srv1608941952:90 , srv1614225351:90 , srv1711473201:90 , srv1789334003:90 , srv1840003123:90 , srv1930276033:90 , srv1957955768:90 , srv2032702341:90 , srv2076626413:90 , srv2084542689:90 , srv2121394197:90 , srv2122005178:90 , srv566294053:90 , srv69930587:90 , srv746155638:90 , srv792855642:90 , srv816211670:90 , srv877873678:90 , srv887644144:90 , srv906036428:90 , srv95373157:90 , srv992135118:90 } 2023-09-02 21:55:14,900 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=225 (was 279), ProcessCount=170 (was 170), AvailableMemoryMB=9317 (was 9357)