2023-05-15 23:53:23,129 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-05-15 23:53:23,443 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-05-15 23:53:23,555 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-05-15 23:53:23,622 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-15 23:53:23,631 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-15 23:53:23,650 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=327, ProcessCount=169, AvailableMemoryMB=6690 2023-05-15 23:53:23,652 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-15 23:53:23,653 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-05-15 23:53:23,805 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1013216998:84 , srv1079426274:84 , srv1101526153:84 , srv1125222037:84 , srv1153276318:84 , srv1180210074:84 , srv1191156819:84 , srv1208957340:84 , srv1209213323:84 , srv1209614298:84 , srv1259745708:84 , srv1310581905:84 , srv132114889:84 , srv1339447003:84 , srv1366677186:84 , srv138891987:84 , srv1389847441:84 , srv1397317970:84 , srv1440456019:84 , srv1448054297:84 , srv1454074474:84 , srv1461234713:84 , srv1501056257:84 , srv1505436112:84 , srv1577703928:84 , srv1580981528:84 , srv1582808964:84 , srv1585100048:84 , srv1590287157:84 , srv1597692076:84 , srv1615174148:84 , srv1686103421:84 , srv1688694521:84 , srv1692607017:84 , srv1697017863:84 , srv1702663203:84 , srv1712957155:84 , srv1747563662:84 , srv177622782:84 , srv1787051451:84 , srv1788981668:84 , srv1793580630:84 , srv180157876:84 , srv1827340235:84 , srv1855075107:84 , srv1858616434:84 , srv1870918517:84 , srv1899421239:84 , srv1917132855:84 , srv1925294736:84 , srv1961265630:84 , srv1977964083:84 , srv1991035169:84 , srv1993824475:84 , srv2019334518:84 , srv2038539776:84 , srv2068855682:84 , srv2094445040:84 , srv2099383203:84 , srv2111816243:84 , srv2121315750:84 , srv215748801:84 , srv224217364:84 , srv26689902:84 , srv280120993:84 , srv288641131:84 , srv354990872:84 , srv410043786:84 , srv473929264:84 , srv494981753:84 , srv564327358:84 , srv576962052:84 , srv588603294:84 , srv592848478:84 , srv610755722:84 , srv642163772:84 , srv68369599:84 , srv696380785:84 , srv714178409:84 , srv743121525:84 , srv750949879:84 , srv753825559:84 , srv760437576:84 , srv765221166:84 , srv78563972:84 , srv798924022:84 , srv799798794:84 , srv803570206:84 , srv838844016:84 , srv840430237:84 , srv852723003:84 , srv873064633:84 , srv876842893:84 , srv924437346:84 , srv929851751:84 , srv937770680:84 , srv967065951:84 , srv971654752:84 , srv994334533:84 , srv1609627217:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-05-15 23:53:23,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 23:53:23,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2094445040=58, srv1259745708=10, srv1615174148=31, srv132114889=12, srv180157876=43, srv803570206=88, srv2111816243=60, srv2038539776=56, srv760437576=83, srv592848478=74, srv1590287157=28, srv1899421239=48, srv2121315750=61, srv1448054297=19, srv1580981528=25, srv743121525=80, srv280120993=65, srv610755722=75, srv2099383203=59, srv765221166=84, srv1793580630=42, srv1993824475=54, srv1597692076=29, srv1917132855=49, srv971654752=98, srv1101526153=2, srv642163772=76, srv838844016=89, srv1461234713=21, srv576962052=72, srv1585100048=27, srv1454074474=20, srv924437346=94, srv929851751=95, srv494981753=70, srv1209213323=8, srv1787051451=40, srv753825559=82, srv1582808964=26, srv750949879=81, srv994334533=99, srv876842893=93, srv2068855682=57, srv696380785=78, srv564327358=71, srv1870918517=47, srv1208957340=7, srv1310581905=11, srv1827340235=44, srv1961265630=51, srv354990872=67, srv1013216998=0, srv840430237=90, srv798924022=86, srv1977964083=52, srv224217364=63, srv1788981668=41, srv1855075107=45, srv1505436112=23, srv1191156819=6, srv1153276318=4, srv1712957155=37, srv68369599=77, srv1389847441=16, srv1501056257=22, srv1925294736=50, srv1079426274=1, srv588603294=73, srv852723003=91, srv2019334518=55, srv1339447003=13, srv78563972=85, srv1209614298=9, srv26689902=64, srv288641131=66, srv967065951=97, srv1440456019=18, srv1577703928=24, srv1702663203=36, srv873064633=92, srv799798794=87, srv1858616434=46, srv1686103421=32, srv177622782=39, srv1366677186=14, srv714178409=79, srv937770680=96, srv1125222037=3, srv410043786=68, srv1991035169=53, srv473929264=69, srv1397317970=17, srv215748801=62, srv1692607017=34, srv1697017863=35, srv1180210074=5, srv1609627217=30, srv1747563662=38, srv138891987=15, srv1688694521=33} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 23:53:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 23:53:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 23:53:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 23:53:23,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 23:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 23:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 23:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 23:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 23:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 23:53:23,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 23:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 23:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 23:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 23:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 23:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 23:53:23,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 23:53:23,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 23:53:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 23:53:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 23:53:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 23:53:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 23:53:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 23:53:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 23:53:23,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 23:53:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-15 23:53:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-15 23:53:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-15 23:53:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-15 23:53:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-15 23:53:23,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-15 23:53:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-15 23:53:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-15 23:53:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-15 23:53:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-15 23:53:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-15 23:53:23,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-15 23:53:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-15 23:53:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-15 23:53:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-15 23:53:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-15 23:53:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-15 23:53:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-15 23:53:23,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-15 23:53:23,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-15 23:53:23,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-15 23:53:23,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-15 23:53:23,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-15 23:53:23,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-15 23:53:23,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-15 23:53:23,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-15 23:53:23,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-15 23:53:23,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-15 23:53:23,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-15 23:53:23,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-15 23:53:23,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-15 23:53:23,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-15 23:53:23,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-15 23:53:23,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-15 23:53:23,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-15 23:53:23,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-15 23:53:23,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-15 23:53:23,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-15 23:53:23,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-15 23:53:23,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-15 23:53:23,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-15 23:53:23,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-15 23:53:23,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-15 23:53:23,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-15 23:53:23,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-15 23:53:23,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-15 23:53:23,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-15 23:53:23,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-15 23:53:23,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-15 23:53:23,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-15 23:53:23,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-15 23:53:23,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-15 23:53:23,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-15 23:53:23,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-15 23:53:23,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-15 23:53:23,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-15 23:53:23,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-15 23:53:23,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-15 23:53:23,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-15 23:53:23,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-15 23:53:23,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-15 23:53:23,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-15 23:53:23,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-15 23:53:23,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-15 23:53:23,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-15 23:53:23,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-15 23:53:23,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-15 23:53:23,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-15 23:53:23,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-15 23:53:23,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-15 23:53:23,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-15 23:53:24,252 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 23:53:24,252 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-15 23:53:24,252 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-15 23:53:24,618 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 651 ms to try 5000 different iterations. Found a solution that moves 4677 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1825278777665799. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03939180977869135, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053206920893587056, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5196666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03939180977869135, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17791666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24316666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 23:53:24,641 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1615174148:82 , srv1702663203:82 , srv588603294:82 , srv799798794:82 , srv1101526153:83 , srv1180210074:83 , srv1686103421:83 , srv1855075107:83 , srv1870918517:83 , srv2111816243:83 , srv803570206:83 , srv1079426274:84 , srv1153276318:84 , srv1582808964:84 , srv1585100048:84 , srv1688694521:84 , srv1697017863:84 , srv1747563662:84 , srv1977964083:84 , srv1993824475:84 , srv288641131:84 , srv610755722:84 , srv760437576:84 , srv1208957340:85 , srv1259745708:85 , srv1366677186:85 , srv1440456019:85 , srv1577703928:85 , srv1580981528:85 , srv1590287157:85 , srv1925294736:85 , srv1991035169:85 , srv2068855682:85 , srv2094445040:85 , srv2099383203:85 , srv280120993:85 , srv354990872:85 , srv410043786:85 , srv473929264:85 , srv68369599:85 , srv743121525:85 , srv838844016:85 , srv876842893:85 , srv1209213323:86 , srv1310581905:86 , srv1448054297:86 , srv1712957155:86 , srv177622782:86 , srv1858616434:86 , srv2121315750:86 , srv26689902:86 , srv750949879:86 , srv937770680:86 , srv1013216998:87 , srv1125222037:87 , srv1209614298:87 , srv1397317970:87 , srv1917132855:87 , srv2019334518:87 , srv2038539776:87 , srv215748801:87 , srv576962052:87 , srv696380785:87 , srv765221166:87 , srv798924022:87 , srv852723003:87 , srv924437346:87 , srv967065951:87 , srv132114889:88 , srv1339447003:88 , srv138891987:88 , srv1597692076:88 , srv1788981668:88 , srv1827340235:88 , srv494981753:88 , srv564327358:88 , srv753825559:88 , srv1461234713:89 , srv1505436112:89 , srv1692607017:89 , srv1899421239:89 , srv1961265630:89 , srv592848478:89 , srv714178409:89 , srv78563972:89 , srv873064633:89 , srv929851751:89 , srv994334533:89 , srv1191156819:90 , srv1501056257:90 , srv1787051451:90 , srv971654752:90 , srv642163772:91 , srv840430237:91 , srv1389847441:92 , srv1454074474:92 , srv180157876:92 , srv224217364:93 , srv1793580630:96 , srv1609627217:440 } 2023-05-15 23:53:24,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 23:53:24,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2094445040=58, srv1259745708=10, srv1615174148=31, srv132114889=12, srv180157876=43, srv803570206=88, srv2111816243=60, srv2038539776=56, srv760437576=83, srv592848478=74, srv1590287157=28, srv1899421239=48, srv2121315750=61, srv1448054297=19, srv1580981528=25, srv743121525=80, srv280120993=65, srv610755722=75, srv2099383203=59, srv765221166=84, srv1793580630=42, srv1993824475=54, srv1597692076=29, srv1917132855=49, srv971654752=98, srv1101526153=2, srv642163772=76, srv838844016=89, srv1461234713=21, srv576962052=72, srv1585100048=27, srv1454074474=20, srv924437346=94, srv929851751=95, srv494981753=70, srv1209213323=8, srv1787051451=40, srv753825559=82, srv1582808964=26, srv750949879=81, srv994334533=99, srv876842893=93, srv2068855682=57, srv696380785=78, srv564327358=71, srv1870918517=47, srv1208957340=7, srv1310581905=11, srv1827340235=44, srv1961265630=51, srv354990872=67, srv1013216998=0, srv840430237=90, srv798924022=86, srv1977964083=52, srv224217364=63, srv1788981668=41, srv1855075107=45, srv1505436112=23, srv1191156819=6, srv1153276318=4, srv1712957155=37, srv68369599=77, srv1389847441=16, srv1501056257=22, srv1925294736=50, srv1079426274=1, srv588603294=73, srv852723003=91, srv2019334518=55, srv1339447003=13, srv78563972=85, srv1209614298=9, srv26689902=64, srv288641131=66, srv967065951=97, srv1440456019=18, srv1577703928=24, srv1702663203=36, srv873064633=92, srv799798794=87, srv1858616434=46, srv1686103421=32, srv177622782=39, srv1366677186=14, srv714178409=79, srv937770680=96, srv1125222037=3, srv410043786=68, srv1991035169=53, srv473929264=69, srv1397317970=17, srv215748801=62, srv1692607017=34, srv1697017863=35, srv1180210074=5, srv1609627217=30, srv1747563662=38, srv138891987=15, srv1688694521=33} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 23:53:24,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 23:53:24,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 23:53:24,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 23:53:24,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 23:53:24,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 23:53:24,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 23:53:24,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 23:53:24,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 23:53:24,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 23:53:24,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 23:53:24,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 23:53:24,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 23:53:24,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 23:53:24,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 23:53:24,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 23:53:24,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 23:53:24,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 23:53:24,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 23:53:24,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 23:53:24,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 23:53:24,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 23:53:24,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 23:53:24,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 23:53:24,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 23:53:24,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 23:53:24,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 23:53:24,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 23:53:24,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 23:53:24,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 23:53:24,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 23:53:24,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-15 23:53:24,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-15 23:53:24,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-15 23:53:24,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-15 23:53:24,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-15 23:53:24,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-15 23:53:24,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-15 23:53:24,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-15 23:53:24,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-15 23:53:24,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-15 23:53:24,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-15 23:53:24,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-15 23:53:24,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-15 23:53:24,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-15 23:53:24,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-15 23:53:24,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-15 23:53:24,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-15 23:53:24,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-15 23:53:24,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-15 23:53:24,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-15 23:53:24,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-15 23:53:24,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-15 23:53:24,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-15 23:53:24,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-15 23:53:24,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-15 23:53:24,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-15 23:53:24,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-15 23:53:24,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-15 23:53:24,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-15 23:53:24,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-15 23:53:24,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-15 23:53:24,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-15 23:53:24,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-15 23:53:24,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-15 23:53:24,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-15 23:53:24,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-15 23:53:24,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-15 23:53:24,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-15 23:53:24,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-15 23:53:24,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-15 23:53:24,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-15 23:53:24,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-15 23:53:24,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-15 23:53:24,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-15 23:53:24,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-15 23:53:24,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-15 23:53:24,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-15 23:53:24,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-15 23:53:24,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-15 23:53:24,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-15 23:53:24,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-15 23:53:24,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-15 23:53:24,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-15 23:53:24,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-15 23:53:24,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-15 23:53:24,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-15 23:53:24,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-15 23:53:24,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-15 23:53:24,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-15 23:53:24,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-15 23:53:24,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-15 23:53:24,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-15 23:53:24,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-15 23:53:24,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-15 23:53:24,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-15 23:53:24,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-15 23:53:24,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-15 23:53:24,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-15 23:53:24,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-15 23:53:24,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-15 23:53:24,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-15 23:53:24,867 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03939180977869135, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053206920893587056, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03939180977869135, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17791666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24316666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 23:53:24,867 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-15 23:53:24,868 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18249512871262033, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03939180977869135, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053206920893587056, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03939180977869135, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17791666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24316666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-15 23:53:25,174 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 492 ms to try 5000 different iterations. Found a solution that moves 2647 regions; Going from a computed imbalance of 0.18249512871262033 to a new imbalance of 0.01587072982231231. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022048705283166246); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031369776070758255, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2941111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022048705283166246); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.045, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 23:53:25,187 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1793580630:78 , srv1191156819:79 , srv1389847441:80 , srv1454074474:81 , srv180157876:81 , srv224217364:81 , srv1505436112:82 , srv1961265630:82 , srv494981753:82 , srv642163772:82 , srv873064633:82 , srv924437346:82 , srv929851751:82 , srv1397317970:83 , srv1501056257:83 , srv1597692076:83 , srv1692607017:83 , srv1899421239:83 , srv753825559:83 , srv971654752:83 , srv994334533:83 , srv132114889:84 , srv1461234713:84 , srv177622782:84 , srv1787051451:84 , srv2038539776:84 , srv26689902:84 , srv564327358:84 , srv576962052:84 , srv592848478:84 , srv714178409:84 , srv78563972:84 , srv840430237:84 , srv1208957340:85 , srv1209614298:85 , srv1712957155:85 , srv1788981668:85 , srv1827340235:85 , srv2019334518:85 , srv473929264:85 , srv696380785:85 , srv765221166:85 , srv798924022:85 , srv937770680:85 , srv1013216998:86 , srv1125222037:86 , srv1209213323:86 , srv1310581905:86 , srv1339447003:86 , srv138891987:86 , srv1440456019:86 , srv1590287157:86 , srv1697017863:86 , srv1747563662:86 , srv1925294736:86 , srv2099383203:86 , srv2121315750:86 , srv852723003:86 , srv967065951:86 , srv1153276318:87 , srv1259745708:87 , srv1366677186:87 , srv1448054297:87 , srv1580981528:87 , srv1582808964:87 , srv1585100048:87 , srv1688694521:87 , srv1858616434:87 , srv1870918517:87 , srv1993824475:87 , srv2094445040:87 , srv215748801:87 , srv354990872:87 , srv610755722:87 , srv743121525:87 , srv750949879:87 , srv803570206:87 , srv838844016:87 , srv1577703928:88 , srv1686103421:88 , srv1855075107:88 , srv1917132855:88 , srv1977964083:88 , srv1991035169:88 , srv288641131:88 , srv410043786:88 , srv588603294:88 , srv68369599:88 , srv760437576:88 , srv876842893:88 , srv1079426274:89 , srv1615174148:89 , srv280120993:89 , srv799798794:89 , srv2068855682:90 , srv2111816243:90 , srv1101526153:91 , srv1702663203:91 , srv1180210074:92 , srv1609627217:530 } 2023-05-15 23:53:25,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 23:53:25,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2094445040=58, srv1259745708=10, srv1615174148=31, srv132114889=12, srv180157876=43, srv803570206=88, srv2111816243=60, srv2038539776=56, srv760437576=83, srv592848478=74, srv1590287157=28, srv1899421239=48, srv2121315750=61, srv1448054297=19, srv1580981528=25, srv743121525=80, srv280120993=65, srv610755722=75, srv2099383203=59, srv765221166=84, srv1793580630=42, srv1993824475=54, srv1597692076=29, srv1917132855=49, srv971654752=98, srv1101526153=2, srv642163772=76, srv838844016=89, srv1461234713=21, srv576962052=72, srv1585100048=27, srv1454074474=20, srv924437346=94, srv929851751=95, srv494981753=70, srv1209213323=8, srv1787051451=40, srv753825559=82, srv1582808964=26, srv750949879=81, srv994334533=99, srv876842893=93, srv2068855682=57, srv696380785=78, srv564327358=71, srv1870918517=47, srv1208957340=7, srv1310581905=11, srv1827340235=44, srv1961265630=51, srv354990872=67, srv1013216998=0, srv840430237=90, srv798924022=86, srv1977964083=52, srv224217364=63, srv1788981668=41, srv1855075107=45, srv1505436112=23, srv1191156819=6, srv1153276318=4, srv1712957155=37, srv68369599=77, srv1389847441=16, srv1501056257=22, srv1925294736=50, srv1079426274=1, srv588603294=73, srv852723003=91, srv2019334518=55, srv1339447003=13, srv78563972=85, srv1209614298=9, srv26689902=64, srv288641131=66, srv967065951=97, srv1440456019=18, srv1577703928=24, srv1702663203=36, srv873064633=92, srv799798794=87, srv1858616434=46, srv1686103421=32, srv177622782=39, srv1366677186=14, srv714178409=79, srv937770680=96, srv1125222037=3, srv410043786=68, srv1991035169=53, srv473929264=69, srv1397317970=17, srv215748801=62, srv1692607017=34, srv1697017863=35, srv1180210074=5, srv1609627217=30, srv1747563662=38, srv138891987=15, srv1688694521=33} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 23:53:25,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 23:53:25,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 23:53:25,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 23:53:25,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 23:53:25,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 23:53:25,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 23:53:25,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 23:53:25,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 23:53:25,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 23:53:25,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 23:53:25,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 23:53:25,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 23:53:25,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 23:53:25,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 23:53:25,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 23:53:25,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 23:53:25,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 23:53:25,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 23:53:25,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 23:53:25,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 23:53:25,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 23:53:25,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 23:53:25,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 23:53:25,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 23:53:25,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 23:53:25,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 23:53:25,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 23:53:25,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 23:53:25,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 23:53:25,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 23:53:25,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-15 23:53:25,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-15 23:53:25,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-15 23:53:25,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-15 23:53:25,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-15 23:53:25,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-15 23:53:25,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-15 23:53:25,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-15 23:53:25,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-15 23:53:25,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-15 23:53:25,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-15 23:53:25,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-15 23:53:25,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-15 23:53:25,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-15 23:53:25,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-15 23:53:25,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-15 23:53:25,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-15 23:53:25,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-15 23:53:25,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-15 23:53:25,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-15 23:53:25,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-15 23:53:25,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-15 23:53:25,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-15 23:53:25,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-15 23:53:25,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-15 23:53:25,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-15 23:53:25,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-15 23:53:25,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-15 23:53:25,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-15 23:53:25,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-15 23:53:25,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-15 23:53:25,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-15 23:53:25,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-15 23:53:25,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-15 23:53:25,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-15 23:53:25,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-15 23:53:25,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-15 23:53:25,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-15 23:53:25,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-15 23:53:25,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-15 23:53:25,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-15 23:53:25,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-15 23:53:25,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-15 23:53:25,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-15 23:53:25,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-15 23:53:25,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-15 23:53:25,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-15 23:53:25,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-15 23:53:25,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-15 23:53:25,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-15 23:53:25,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-15 23:53:25,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-15 23:53:25,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-15 23:53:25,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-15 23:53:25,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-15 23:53:25,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-15 23:53:25,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-15 23:53:25,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-15 23:53:25,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-15 23:53:25,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-15 23:53:25,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-15 23:53:25,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-15 23:53:25,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-15 23:53:25,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-15 23:53:25,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-15 23:53:25,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-15 23:53:25,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-15 23:53:25,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-15 23:53:25,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-15 23:53:25,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-15 23:53:25,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-15 23:53:25,433 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022048705283166246); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031369776070758255, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022048705283166246); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.045, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 23:53:25,433 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-15 23:53:25,433 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015852195132162433, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022048705283166246); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031369776070758255, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022048705283166246); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.045, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-15 23:53:25,712 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 457 ms to try 5000 different iterations. Found a solution that moves 1023 regions; Going from a computed imbalance of 0.015852195132162433 to a new imbalance of 0.001334502734245334. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006655433858912234); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006085806194501845); MoveCostFunction : (multiplier=7.0, imbalance=0.11366666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006655433858912234); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=9.166666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 23:53:25,714 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1180210074:82 , srv138891987:83 , srv1787051451:83 , srv1917132855:83 , srv280120993:83 , srv840430237:83 , srv1013216998:84 , srv1101526153:84 , srv1125222037:84 , srv1339447003:84 , srv1448054297:84 , srv1454074474:84 , srv1577703928:84 , srv1702663203:84 , srv1793580630:84 , srv1858616434:84 , srv2068855682:84 , srv215748801:84 , srv224217364:84 , srv410043786:84 , srv68369599:84 , srv714178409:84 , srv750949879:84 , srv78563972:84 , srv852723003:84 , srv1079426274:85 , srv1209213323:85 , srv1259745708:85 , srv132114889:85 , srv1366677186:85 , srv1389847441:85 , srv1461234713:85 , srv1501056257:85 , srv1585100048:85 , srv1590287157:85 , srv1692607017:85 , srv1788981668:85 , srv180157876:85 , srv1827340235:85 , srv1977964083:85 , srv1991035169:85 , srv2019334518:85 , srv2094445040:85 , srv2111816243:85 , srv288641131:85 , srv354990872:85 , srv564327358:85 , srv592848478:85 , srv696380785:85 , srv743121525:85 , srv765221166:85 , srv798924022:85 , srv876842893:85 , srv967065951:85 , srv971654752:85 , srv994334533:85 , srv1153276318:86 , srv1310581905:86 , srv1440456019:86 , srv1505436112:86 , srv1580981528:86 , srv1582808964:86 , srv1686103421:86 , srv1712957155:86 , srv1855075107:86 , srv1899421239:86 , srv1925294736:86 , srv1993824475:86 , srv2038539776:86 , srv2099383203:86 , srv2121315750:86 , srv576962052:86 , srv642163772:86 , srv753825559:86 , srv760437576:86 , srv838844016:86 , srv929851751:86 , srv1208957340:87 , srv1209614298:87 , srv1615174148:87 , srv1688694521:87 , srv1697017863:87 , srv1747563662:87 , srv177622782:87 , srv1870918517:87 , srv1961265630:87 , srv26689902:87 , srv473929264:87 , srv588603294:87 , srv610755722:87 , srv799798794:87 , srv803570206:87 , srv873064633:87 , srv937770680:87 , srv1191156819:88 , srv1397317970:88 , srv1597692076:88 , srv494981753:88 , srv924437346:88 , srv1609627217:547 } 2023-05-15 23:53:25,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 23:53:25,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2094445040=58, srv1259745708=10, srv1615174148=31, srv132114889=12, srv180157876=43, srv803570206=88, srv2111816243=60, srv2038539776=56, srv760437576=83, srv592848478=74, srv1590287157=28, srv1899421239=48, srv2121315750=61, srv1448054297=19, srv1580981528=25, srv743121525=80, srv280120993=65, srv610755722=75, srv2099383203=59, srv765221166=84, srv1793580630=42, srv1993824475=54, srv1597692076=29, srv1917132855=49, srv971654752=98, srv1101526153=2, srv642163772=76, srv838844016=89, srv1461234713=21, srv576962052=72, srv1585100048=27, srv1454074474=20, srv924437346=94, srv929851751=95, srv494981753=70, srv1209213323=8, srv1787051451=40, srv753825559=82, srv1582808964=26, srv750949879=81, srv994334533=99, srv876842893=93, srv2068855682=57, srv696380785=78, srv564327358=71, srv1870918517=47, srv1208957340=7, srv1310581905=11, srv1827340235=44, srv1961265630=51, srv354990872=67, srv1013216998=0, srv840430237=90, srv798924022=86, srv1977964083=52, srv224217364=63, srv1788981668=41, srv1855075107=45, srv1505436112=23, srv1191156819=6, srv1153276318=4, srv1712957155=37, srv68369599=77, srv1389847441=16, srv1501056257=22, srv1925294736=50, srv1079426274=1, srv588603294=73, srv852723003=91, srv2019334518=55, srv1339447003=13, srv78563972=85, srv1209614298=9, srv26689902=64, srv288641131=66, srv967065951=97, srv1440456019=18, srv1577703928=24, srv1702663203=36, srv873064633=92, srv799798794=87, srv1858616434=46, srv1686103421=32, srv177622782=39, srv1366677186=14, srv714178409=79, srv937770680=96, srv1125222037=3, srv410043786=68, srv1991035169=53, srv473929264=69, srv1397317970=17, srv215748801=62, srv1692607017=34, srv1697017863=35, srv1180210074=5, srv1609627217=30, srv1747563662=38, srv138891987=15, srv1688694521=33} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 23:53:25,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 23:53:25,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 23:53:25,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 23:53:25,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 23:53:25,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 23:53:25,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 23:53:25,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 23:53:25,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 23:53:25,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 23:53:25,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 23:53:25,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 23:53:25,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 23:53:25,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 23:53:25,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 23:53:25,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 23:53:25,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 23:53:25,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 23:53:25,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 23:53:25,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 23:53:25,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 23:53:25,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 23:53:25,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 23:53:25,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 23:53:25,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 23:53:25,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 23:53:25,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 23:53:25,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 23:53:25,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 23:53:25,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 23:53:25,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 23:53:25,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-15 23:53:25,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-15 23:53:25,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-15 23:53:25,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-15 23:53:25,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-15 23:53:25,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-15 23:53:25,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-15 23:53:25,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-15 23:53:25,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-15 23:53:25,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-15 23:53:25,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-15 23:53:25,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-15 23:53:25,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-15 23:53:25,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-15 23:53:25,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-15 23:53:25,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-15 23:53:25,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-15 23:53:25,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-15 23:53:25,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-15 23:53:25,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-15 23:53:25,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-15 23:53:25,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-15 23:53:25,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-15 23:53:25,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-15 23:53:25,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-15 23:53:25,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-15 23:53:25,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-15 23:53:25,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-15 23:53:25,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-15 23:53:25,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-15 23:53:25,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-15 23:53:25,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-15 23:53:25,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-15 23:53:25,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-15 23:53:25,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-15 23:53:25,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-15 23:53:25,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-15 23:53:25,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-15 23:53:25,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-15 23:53:25,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-15 23:53:25,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-15 23:53:25,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-15 23:53:25,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-15 23:53:25,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-15 23:53:25,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-15 23:53:25,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-15 23:53:25,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-15 23:53:25,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-15 23:53:25,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-15 23:53:25,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-15 23:53:25,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-15 23:53:25,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-15 23:53:25,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-15 23:53:25,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-15 23:53:25,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-15 23:53:25,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-15 23:53:25,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-15 23:53:25,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-15 23:53:25,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-15 23:53:25,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-15 23:53:25,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-15 23:53:25,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-15 23:53:25,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-15 23:53:25,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-15 23:53:25,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-15 23:53:25,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-15 23:53:25,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-15 23:53:25,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-15 23:53:25,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-15 23:53:25,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-15 23:53:25,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-15 23:53:25,853 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006655433858912234); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006085806194501845); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006655433858912234); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=9.166666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 23:53:25,853 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-15 23:53:25,854 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0013273395351432096, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006655433858912234); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006085806194501845); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006655433858912234); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=9.166666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-15 23:53:26,326 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 578 ms to try 5000 different iterations. Found a solution that moves 261 regions; Going from a computed imbalance of 0.0013273395351432096 to a new imbalance of 2.8986679835364168E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.029, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 23:53:26,327 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1597692076:83 , srv1615174148:83 , srv2068855682:83 , srv642163772:83 , srv1079426274:84 , srv1101526153:84 , srv1209614298:84 , srv1259745708:84 , srv1310581905:84 , srv132114889:84 , srv1339447003:84 , srv1397317970:84 , srv1461234713:84 , srv1501056257:84 , srv1580981528:84 , srv1582808964:84 , srv1688694521:84 , srv1747563662:84 , srv1788981668:84 , srv1793580630:84 , srv180157876:84 , srv1827340235:84 , srv1899421239:84 , srv1917132855:84 , srv1961265630:84 , srv2111816243:84 , srv2121315750:84 , srv215748801:84 , srv26689902:84 , srv494981753:84 , srv592848478:84 , srv610755722:84 , srv765221166:84 , srv798924022:84 , srv799798794:84 , srv838844016:84 , srv840430237:84 , srv873064633:84 , srv876842893:84 , srv937770680:84 , srv967065951:84 , srv1013216998:85 , srv1125222037:85 , srv1153276318:85 , srv1180210074:85 , srv1191156819:85 , srv1208957340:85 , srv1209213323:85 , srv1366677186:85 , srv138891987:85 , srv1389847441:85 , srv1440456019:85 , srv1448054297:85 , srv1454074474:85 , srv1505436112:85 , srv1577703928:85 , srv1686103421:85 , srv1692607017:85 , srv1697017863:85 , srv1702663203:85 , srv1712957155:85 , srv177622782:85 , srv1787051451:85 , srv1855075107:85 , srv1858616434:85 , srv1870918517:85 , srv1925294736:85 , srv1977964083:85 , srv1991035169:85 , srv1993824475:85 , srv2019334518:85 , srv2038539776:85 , srv2094445040:85 , srv2099383203:85 , srv224217364:85 , srv280120993:85 , srv288641131:85 , srv354990872:85 , srv410043786:85 , srv473929264:85 , srv564327358:85 , srv576962052:85 , srv588603294:85 , srv68369599:85 , srv696380785:85 , srv714178409:85 , srv743121525:85 , srv750949879:85 , srv753825559:85 , srv760437576:85 , srv803570206:85 , srv852723003:85 , srv924437346:85 , srv971654752:85 , srv1585100048:86 , srv1590287157:86 , srv78563972:86 , srv929851751:86 , srv994334533:86 , srv1609627217:625 } 2023-05-15 23:53:26,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 23:53:26,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2094445040=58, srv1259745708=10, srv1615174148=31, srv132114889=12, srv180157876=43, srv803570206=88, srv2111816243=60, srv2038539776=56, srv760437576=83, srv592848478=74, srv1590287157=28, srv1899421239=48, srv2121315750=61, srv1448054297=19, srv1580981528=25, srv743121525=80, srv280120993=65, srv610755722=75, srv2099383203=59, srv765221166=84, srv1793580630=42, srv1993824475=54, srv1597692076=29, srv1917132855=49, srv971654752=98, srv1101526153=2, srv642163772=76, srv838844016=89, srv1461234713=21, srv576962052=72, srv1585100048=27, srv1454074474=20, srv924437346=94, srv929851751=95, srv494981753=70, srv1209213323=8, srv1787051451=40, srv753825559=82, srv1582808964=26, srv750949879=81, srv994334533=99, srv876842893=93, srv2068855682=57, srv696380785=78, srv564327358=71, srv1870918517=47, srv1208957340=7, srv1310581905=11, srv1827340235=44, srv1961265630=51, srv354990872=67, srv1013216998=0, srv840430237=90, srv798924022=86, srv1977964083=52, srv224217364=63, srv1788981668=41, srv1855075107=45, srv1505436112=23, srv1191156819=6, srv1153276318=4, srv1712957155=37, srv68369599=77, srv1389847441=16, srv1501056257=22, srv1925294736=50, srv1079426274=1, srv588603294=73, srv852723003=91, srv2019334518=55, srv1339447003=13, srv78563972=85, srv1209614298=9, srv26689902=64, srv288641131=66, srv967065951=97, srv1440456019=18, srv1577703928=24, srv1702663203=36, srv873064633=92, srv799798794=87, srv1858616434=46, srv1686103421=32, srv177622782=39, srv1366677186=14, srv714178409=79, srv937770680=96, srv1125222037=3, srv410043786=68, srv1991035169=53, srv473929264=69, srv1397317970=17, srv215748801=62, srv1692607017=34, srv1697017863=35, srv1180210074=5, srv1609627217=30, srv1747563662=38, srv138891987=15, srv1688694521=33} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 23:53:26,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 23:53:26,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 23:53:26,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 23:53:26,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 23:53:26,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 23:53:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 23:53:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 23:53:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 23:53:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 23:53:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 23:53:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 23:53:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 23:53:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 23:53:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 23:53:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 23:53:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 23:53:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 23:53:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 23:53:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 23:53:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 23:53:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 23:53:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 23:53:26,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 23:53:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 23:53:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 23:53:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 23:53:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 23:53:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 23:53:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 23:53:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 23:53:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-15 23:53:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-15 23:53:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-15 23:53:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-15 23:53:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-15 23:53:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-15 23:53:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-15 23:53:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-15 23:53:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-15 23:53:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-15 23:53:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-15 23:53:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-15 23:53:26,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-15 23:53:26,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-15 23:53:26,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-15 23:53:26,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-15 23:53:26,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-15 23:53:26,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-15 23:53:26,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-15 23:53:26,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-15 23:53:26,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-15 23:53:26,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-15 23:53:26,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-15 23:53:26,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-15 23:53:26,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-15 23:53:26,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-15 23:53:26,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-15 23:53:26,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-15 23:53:26,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-15 23:53:26,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-15 23:53:26,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-15 23:53:26,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-15 23:53:26,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-15 23:53:26,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-15 23:53:26,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-15 23:53:26,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-15 23:53:26,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-15 23:53:26,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-15 23:53:26,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-15 23:53:26,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-15 23:53:26,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-15 23:53:26,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-15 23:53:26,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-15 23:53:26,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-15 23:53:26,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-15 23:53:26,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-15 23:53:26,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-15 23:53:26,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-15 23:53:26,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-15 23:53:26,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-15 23:53:26,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-15 23:53:26,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-15 23:53:26,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-15 23:53:26,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-15 23:53:26,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-15 23:53:26,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-15 23:53:26,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-15 23:53:26,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-15 23:53:26,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-15 23:53:26,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-15 23:53:26,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-15 23:53:26,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-15 23:53:26,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-15 23:53:26,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-15 23:53:26,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-15 23:53:26,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-15 23:53:26,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-15 23:53:26,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-15 23:53:26,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-15 23:53:26,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-15 23:53:26,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-15 23:53:26,487 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 23:53:26,487 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-05-15 23:53:26,488 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.7159118774118367E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-05-15 23:53:26,774 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 413 ms to try 5000 different iterations. Found a solution that moves 36 regions; Going from a computed imbalance of 2.7159118774118367E-5 to a new imbalance of 6.650063060027741E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.004); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 23:53:26,775 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1209614298:83 , srv1991035169:83 , srv224217364:83 , srv760437576:83 , srv78563972:83 , srv929851751:83 , srv971654752:83 , srv994334533:83 , srv1013216998:84 , srv1079426274:84 , srv1101526153:84 , srv1125222037:84 , srv1153276318:84 , srv1180210074:84 , srv1191156819:84 , srv1208957340:84 , srv1209213323:84 , srv1310581905:84 , srv1339447003:84 , srv1366677186:84 , srv138891987:84 , srv1389847441:84 , srv1397317970:84 , srv1440456019:84 , srv1448054297:84 , srv1454074474:84 , srv1461234713:84 , srv1501056257:84 , srv1505436112:84 , srv1577703928:84 , srv1580981528:84 , srv1585100048:84 , srv1590287157:84 , srv1597692076:84 , srv1686103421:84 , srv1688694521:84 , srv1692607017:84 , srv1697017863:84 , srv1702663203:84 , srv1712957155:84 , srv177622782:84 , srv1787051451:84 , srv1788981668:84 , srv1793580630:84 , srv180157876:84 , srv1827340235:84 , srv1855075107:84 , srv1858616434:84 , srv1870918517:84 , srv1899421239:84 , srv1917132855:84 , srv1925294736:84 , srv1961265630:84 , srv1977964083:84 , srv1993824475:84 , srv2019334518:84 , srv2038539776:84 , srv2068855682:84 , srv2094445040:84 , srv2099383203:84 , srv2111816243:84 , srv2121315750:84 , srv215748801:84 , srv280120993:84 , srv288641131:84 , srv354990872:84 , srv410043786:84 , srv473929264:84 , srv494981753:84 , srv564327358:84 , srv576962052:84 , srv588603294:84 , srv592848478:84 , srv610755722:84 , srv642163772:84 , srv68369599:84 , srv696380785:84 , srv714178409:84 , srv743121525:84 , srv750949879:84 , srv753825559:84 , srv799798794:84 , srv803570206:84 , srv838844016:84 , srv840430237:84 , srv852723003:84 , srv873064633:84 , srv876842893:84 , srv924437346:84 , srv937770680:84 , srv967065951:84 , srv1259745708:85 , srv132114889:85 , srv1582808964:85 , srv1615174148:85 , srv1747563662:85 , srv26689902:85 , srv765221166:85 , srv798924022:85 , srv1609627217:684 } 2023-05-15 23:53:26,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 23:53:26,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2094445040=58, srv1259745708=10, srv1615174148=31, srv132114889=12, srv180157876=43, srv803570206=88, srv2111816243=60, srv2038539776=56, srv760437576=83, srv592848478=74, srv1590287157=28, srv1899421239=48, srv2121315750=61, srv1448054297=19, srv1580981528=25, srv743121525=80, srv280120993=65, srv610755722=75, srv2099383203=59, srv765221166=84, srv1793580630=42, srv1993824475=54, srv1597692076=29, srv1917132855=49, srv971654752=98, srv1101526153=2, srv642163772=76, srv838844016=89, srv1461234713=21, srv576962052=72, srv1585100048=27, srv1454074474=20, srv924437346=94, srv929851751=95, srv494981753=70, srv1209213323=8, srv1787051451=40, srv753825559=82, srv1582808964=26, srv750949879=81, srv994334533=99, srv876842893=93, srv2068855682=57, srv696380785=78, srv564327358=71, srv1870918517=47, srv1208957340=7, srv1310581905=11, srv1827340235=44, srv1961265630=51, srv354990872=67, srv1013216998=0, srv840430237=90, srv798924022=86, srv1977964083=52, srv224217364=63, srv1788981668=41, srv1855075107=45, srv1505436112=23, srv1191156819=6, srv1153276318=4, srv1712957155=37, srv68369599=77, srv1389847441=16, srv1501056257=22, srv1925294736=50, srv1079426274=1, srv588603294=73, srv852723003=91, srv2019334518=55, srv1339447003=13, srv78563972=85, srv1209614298=9, srv26689902=64, srv288641131=66, srv967065951=97, srv1440456019=18, srv1577703928=24, srv1702663203=36, srv873064633=92, srv799798794=87, srv1858616434=46, srv1686103421=32, srv177622782=39, srv1366677186=14, srv714178409=79, srv937770680=96, srv1125222037=3, srv410043786=68, srv1991035169=53, srv473929264=69, srv1397317970=17, srv215748801=62, srv1692607017=34, srv1697017863=35, srv1180210074=5, srv1609627217=30, srv1747563662=38, srv138891987=15, srv1688694521=33} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 23:53:26,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 23:53:26,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 23:53:26,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 23:53:26,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 23:53:26,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 23:53:26,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 23:53:26,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 23:53:26,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 23:53:26,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 23:53:26,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 23:53:26,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 23:53:26,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 23:53:26,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 23:53:26,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 23:53:26,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 23:53:26,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 23:53:26,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 23:53:26,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 23:53:26,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 23:53:26,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 23:53:26,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 23:53:26,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 23:53:26,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 23:53:26,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 23:53:26,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 23:53:26,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 23:53:26,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 23:53:26,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 23:53:26,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 23:53:26,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 23:53:26,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-05-15 23:53:26,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-05-15 23:53:26,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-05-15 23:53:26,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-05-15 23:53:26,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-05-15 23:53:26,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-05-15 23:53:26,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-05-15 23:53:26,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-05-15 23:53:26,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-05-15 23:53:26,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-05-15 23:53:26,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-05-15 23:53:26,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-05-15 23:53:26,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-05-15 23:53:26,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-05-15 23:53:26,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-05-15 23:53:26,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-05-15 23:53:26,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-05-15 23:53:26,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-05-15 23:53:26,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-05-15 23:53:26,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-05-15 23:53:26,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-05-15 23:53:26,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-05-15 23:53:26,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-05-15 23:53:26,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-05-15 23:53:26,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-05-15 23:53:26,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-05-15 23:53:26,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-05-15 23:53:26,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-05-15 23:53:26,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-05-15 23:53:26,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-05-15 23:53:26,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-05-15 23:53:26,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-05-15 23:53:26,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-05-15 23:53:26,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-05-15 23:53:26,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-05-15 23:53:26,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-05-15 23:53:26,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-05-15 23:53:26,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-05-15 23:53:26,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-05-15 23:53:26,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-05-15 23:53:26,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-05-15 23:53:26,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-05-15 23:53:26,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-05-15 23:53:26,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-05-15 23:53:26,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-05-15 23:53:26,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-05-15 23:53:26,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-05-15 23:53:26,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-05-15 23:53:26,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-05-15 23:53:26,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-05-15 23:53:26,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-05-15 23:53:26,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-05-15 23:53:26,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-05-15 23:53:26,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-05-15 23:53:26,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-05-15 23:53:26,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-05-15 23:53:26,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-05-15 23:53:26,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-05-15 23:53:26,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-05-15 23:53:26,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-05-15 23:53:26,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-05-15 23:53:26,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-05-15 23:53:26,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-05-15 23:53:26,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-05-15 23:53:26,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-05-15 23:53:26,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-05-15 23:53:26,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-05-15 23:53:26,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-05-15 23:53:26,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-05-15 23:53:26,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-05-15 23:53:26,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-05-15 23:53:26,959 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.397985672269699E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 23:53:26,960 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1209614298:83 , srv1991035169:83 , srv224217364:83 , srv760437576:83 , srv78563972:83 , srv929851751:83 , srv971654752:83 , srv994334533:83 , srv1013216998:84 , srv1079426274:84 , srv1101526153:84 , srv1125222037:84 , srv1153276318:84 , srv1180210074:84 , srv1191156819:84 , srv1208957340:84 , srv1209213323:84 , srv1310581905:84 , srv1339447003:84 , srv1366677186:84 , srv138891987:84 , srv1389847441:84 , srv1397317970:84 , srv1440456019:84 , srv1448054297:84 , srv1454074474:84 , srv1461234713:84 , srv1501056257:84 , srv1505436112:84 , srv1577703928:84 , srv1580981528:84 , srv1585100048:84 , srv1590287157:84 , srv1597692076:84 , srv1686103421:84 , srv1688694521:84 , srv1692607017:84 , srv1697017863:84 , srv1702663203:84 , srv1712957155:84 , srv177622782:84 , srv1787051451:84 , srv1788981668:84 , srv1793580630:84 , srv180157876:84 , srv1827340235:84 , srv1855075107:84 , srv1858616434:84 , srv1870918517:84 , srv1899421239:84 , srv1917132855:84 , srv1925294736:84 , srv1961265630:84 , srv1977964083:84 , srv1993824475:84 , srv2019334518:84 , srv2038539776:84 , srv2068855682:84 , srv2094445040:84 , srv2099383203:84 , srv2111816243:84 , srv2121315750:84 , srv215748801:84 , srv280120993:84 , srv288641131:84 , srv354990872:84 , srv410043786:84 , srv473929264:84 , srv494981753:84 , srv564327358:84 , srv576962052:84 , srv588603294:84 , srv592848478:84 , srv610755722:84 , srv642163772:84 , srv68369599:84 , srv696380785:84 , srv714178409:84 , srv743121525:84 , srv750949879:84 , srv753825559:84 , srv799798794:84 , srv803570206:84 , srv838844016:84 , srv840430237:84 , srv852723003:84 , srv873064633:84 , srv876842893:84 , srv924437346:84 , srv937770680:84 , srv967065951:84 , srv1259745708:85 , srv132114889:85 , srv1582808964:85 , srv1615174148:85 , srv1747563662:85 , srv26689902:85 , srv765221166:85 , srv798924022:85 , srv1609627217:684 } 2023-05-15 23:53:26,995 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=333 (was 327) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=5837 (was 6690) 2023-05-15 23:53:27,009 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=333, ProcessCount=169, AvailableMemoryMB=5837 2023-05-15 23:53:27,010 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-05-15 23:53:27,011 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-05-15 23:53:27,024 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv108099900:84 , srv1207725159:84 , srv1218316200:84 , srv1229984118:84 , srv1231212054:84 , srv1362380533:84 , srv1415440508:84 , srv1489323952:84 , srv1503021318:84 , srv1503787205:84 , srv1632416013:84 , srv1648473032:84 , srv1661336945:84 , srv1723682096:84 , srv1745953857:84 , srv1822913043:84 , srv1891486198:84 , srv1989618429:84 , srv2001049217:84 , srv2087823523:84 , srv2109090936:84 , srv302296091:84 , srv366131141:84 , srv390314105:84 , srv454588225:84 , srv572579588:84 , srv635833973:84 , srv787468898:84 , srv803213986:84 , srv1238323374:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-05-15 23:53:27,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 23:53:27,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1415440508=7, srv2001049217=19, srv1231212054=4, srv2109090936=21, srv1489323952=8, srv1648473032=12, srv1207725159=1, srv1218316200=2, srv390314105=24, srv1745953857=15, srv1891486198=17, srv635833973=27, srv1503021318=9, srv1661336945=13, srv1229984118=3, srv108099900=0, srv1362380533=6, srv572579588=26, srv1503787205=10, srv302296091=22, srv803213986=29, srv1632416013=11, srv366131141=23, srv1723682096=14, srv454588225=25, srv1989618429=18, srv787468898=28, srv1822913043=16, srv2087823523=20, srv1238323374=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 23:53:27,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 23:53:27,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 23:53:27,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 23:53:27,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 23:53:27,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 23:53:27,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 23:53:27,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 23:53:27,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 23:53:27,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-15 23:53:27,059 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1555724533112275, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 23:53:27,060 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912683246984755, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1555724533112275, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-05-15 23:55:27,041 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5495504 different iterations. Found a solution that moves 1856 regions; Going from a computed imbalance of 0.9912683246984755 to a new imbalance of 4.3319965896196796E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6874074074074074, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 23:55:27,049 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv108099900:90 , srv1207725159:90 , srv1218316200:90 , srv1229984118:90 , srv1231212054:90 , srv1238323374:90 , srv1362380533:90 , srv1415440508:90 , srv1489323952:90 , srv1503021318:90 , srv1503787205:90 , srv1632416013:90 , srv1648473032:90 , srv1661336945:90 , srv1723682096:90 , srv1745953857:90 , srv1822913043:90 , srv1891486198:90 , srv1989618429:90 , srv2001049217:90 , srv2087823523:90 , srv2109090936:90 , srv302296091:90 , srv366131141:90 , srv390314105:90 , srv454588225:90 , srv572579588:90 , srv635833973:90 , srv787468898:90 , srv803213986:90 } 2023-05-15 23:55:27,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-05-15 23:55:27,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1415440508=7, srv2001049217=19, srv1231212054=4, srv2109090936=21, srv1489323952=8, srv1648473032=12, srv1207725159=1, srv1218316200=2, srv390314105=24, srv1745953857=15, srv1891486198=17, srv635833973=27, srv1503021318=9, srv1661336945=13, srv1229984118=3, srv108099900=0, srv1362380533=6, srv572579588=26, srv1503787205=10, srv302296091=22, srv803213986=29, srv1632416013=11, srv366131141=23, srv1723682096=14, srv454588225=25, srv1989618429=18, srv787468898=28, srv1822913043=16, srv2087823523=20, srv1238323374=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-05-15 23:55:27,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-05-15 23:55:27,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-05-15 23:55:27,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-05-15 23:55:27,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-05-15 23:55:27,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-05-15 23:55:27,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-05-15 23:55:27,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-05-15 23:55:27,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-05-15 23:55:27,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-05-15 23:55:27,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-05-15 23:55:27,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-05-15 23:55:27,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-05-15 23:55:27,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-05-15 23:55:27,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-05-15 23:55:27,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-05-15 23:55:27,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-05-15 23:55:27,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-05-15 23:55:27,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-05-15 23:55:27,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-05-15 23:55:27,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-05-15 23:55:27,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-05-15 23:55:27,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-05-15 23:55:27,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-05-15 23:55:27,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-05-15 23:55:27,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-05-15 23:55:27,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-05-15 23:55:27,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-05-15 23:55:27,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-05-15 23:55:27,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-05-15 23:55:27,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-05-15 23:55:27,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-05-15 23:55:27,077 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-05-15 23:55:27,078 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv108099900:90 , srv1207725159:90 , srv1218316200:90 , srv1229984118:90 , srv1231212054:90 , srv1238323374:90 , srv1362380533:90 , srv1415440508:90 , srv1489323952:90 , srv1503021318:90 , srv1503787205:90 , srv1632416013:90 , srv1648473032:90 , srv1661336945:90 , srv1723682096:90 , srv1745953857:90 , srv1822913043:90 , srv1891486198:90 , srv1989618429:90 , srv2001049217:90 , srv2087823523:90 , srv2109090936:90 , srv302296091:90 , srv366131141:90 , srv390314105:90 , srv454588225:90 , srv572579588:90 , srv635833973:90 , srv787468898:90 , srv803213986:90 } 2023-05-15 23:55:27,095 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=276 (was 333), ProcessCount=171 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=6022 (was 5837) - AvailableMemoryMB LEAK? -