2023-02-21 15:53:59,278 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-02-21 15:53:59,618 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-21 15:53:59,735 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-21 15:53:59,798 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-21 15:53:59,807 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-21 15:53:59,822 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=330, ProcessCount=172, AvailableMemoryMB=9769 2023-02-21 15:53:59,824 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-21 15:53:59,825 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-21 15:53:59,952 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1020800256:84 , srv1047805028:84 , srv1086482623:84 , srv1096163443:84 , srv110175828:84 , srv1133778216:84 , srv1149793524:84 , srv1192136238:84 , srv1224366248:84 , srv1268956503:84 , srv1270265089:84 , srv1278928537:84 , srv1280026373:84 , srv12890847:84 , srv1289739963:84 , srv1385708941:84 , srv1396946349:84 , srv1411337809:84 , srv1416367717:84 , srv1449219727:84 , srv1453678693:84 , srv1455900192:84 , srv1473777661:84 , srv1476014151:84 , srv1483194069:84 , srv1509399884:84 , srv1528512911:84 , srv1534882723:84 , srv1538937595:84 , srv1588698834:84 , srv1588861858:84 , srv1626444925:84 , srv1642416006:84 , srv1663160707:84 , srv1671021654:84 , srv1702584796:84 , srv1711271568:84 , srv1728042007:84 , srv173649663:84 , srv1739441805:84 , srv1774729582:84 , srv1775272251:84 , srv1795778349:84 , srv1801434977:84 , srv1816747805:84 , srv1825550077:84 , srv1829016380:84 , srv1831220036:84 , srv1870401905:84 , srv1877369653:84 , srv1887568951:84 , srv1918662982:84 , srv1925978285:84 , srv1947066012:84 , srv194804301:84 , srv1962075914:84 , srv1967703473:84 , srv1990690109:84 , srv1995193228:84 , srv2018046033:84 , srv2036127434:84 , srv2043369321:84 , srv207526673:84 , srv2082294497:84 , srv2125702422:84 , srv2131382946:84 , srv233556544:84 , srv241109374:84 , srv274993801:84 , srv315976577:84 , srv330917977:84 , srv342754561:84 , srv344120215:84 , srv363414844:84 , srv433251492:84 , srv440216289:84 , srv458949034:84 , srv461706654:84 , srv468397816:84 , srv470602390:84 , srv503426113:84 , srv520306952:84 , srv532058390:84 , srv63284980:84 , srv681989867:84 , srv774618466:84 , srv77838584:84 , srv77879529:84 , srv781037534:84 , srv781135591:84 , srv797328769:84 , srv812365250:84 , srv851908634:84 , srv85632200:84 , srv872429626:84 , srv888361000:84 , srv924369892:84 , srv966947036:84 , srv987634221:84 , srv337444073:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-02-21 15:53:59,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-21 15:53:59,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1829016380=46, srv2082294497=63, srv1133778216=5, srv468397816=79, srv1702584796=35, srv1774729582=40, srv1990690109=57, srv851908634=93, srv2018046033=59, srv363414844=74, srv1816747805=44, srv503426113=81, srv330917977=70, srv1671021654=34, srv1270265089=10, srv797328769=91, srv1483194069=24, srv2043369321=61, srv781135591=90, srv1509399884=25, srv77879529=88, srv872429626=95, srv1887568951=50, srv1925978285=52, srv77838584=87, srv85632200=94, srv2036127434=60, srv1289739963=14, srv1962075914=55, srv342754561=72, srv1528512911=26, srv1268956503=9, srv1453678693=20, srv1870401905=48, srv207526673=62, srv888361000=96, srv458949034=77, srv1396946349=16, srv1280026373=12, srv1096163443=3, srv1473777661=22, srv1825550077=45, srv1588698834=29, srv194804301=54, srv315976577=69, srv1047805028=1, srv337444073=71, srv774618466=86, srv440216289=76, srv532058390=83, srv966947036=98, srv1642416006=32, srv1967703473=56, srv233556544=66, srv1538937595=28, srv1801434977=43, srv1711271568=36, srv1278928537=11, srv1020800256=0, srv1224366248=8, srv1831220036=47, srv681989867=85, srv1663160707=33, srv1476014151=23, srv924369892=97, srv63284980=84, srv12890847=13, srv1416367717=18, srv1626444925=31, srv1385708941=15, srv1534882723=27, srv1728042007=37, srv1449219727=19, srv2131382946=65, srv1588861858=30, srv1149793524=6, srv173649663=38, srv1877369653=49, srv781037534=89, srv520306952=82, srv433251492=75, srv110175828=4, srv1192136238=7, srv2125702422=64, srv470602390=80, srv987634221=99, srv1995193228=58, srv1739441805=39, srv1086482623=2, srv1918662982=51, srv461706654=78, srv1947066012=53, srv241109374=67, srv1795778349=42, srv1411337809=17, srv1455900192=21, srv344120215=73, srv1775272251=41, srv812365250=92, srv274993801=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-21 15:54:00,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-21 15:54:00,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-21 15:54:00,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-21 15:54:00,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-21 15:54:00,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-21 15:54:00,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-21 15:54:00,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-21 15:54:00,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-21 15:54:00,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-21 15:54:00,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-21 15:54:00,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-21 15:54:00,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-21 15:54:00,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-21 15:54:00,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-21 15:54:00,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-21 15:54:00,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-21 15:54:00,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-21 15:54:00,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-21 15:54:00,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-21 15:54:00,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-21 15:54:00,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-21 15:54:00,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-21 15:54:00,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-21 15:54:00,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-21 15:54:00,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-21 15:54:00,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-21 15:54:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-21 15:54:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-21 15:54:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-21 15:54:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-21 15:54:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-21 15:54:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-21 15:54:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-21 15:54:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-21 15:54:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-21 15:54:00,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-21 15:54:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-21 15:54:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-21 15:54:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-21 15:54:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-21 15:54:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-21 15:54:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-21 15:54:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-21 15:54:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-21 15:54:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-21 15:54:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-21 15:54:00,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-21 15:54:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-21 15:54:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-21 15:54:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-21 15:54:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-21 15:54:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-21 15:54:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-21 15:54:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-21 15:54:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-21 15:54:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-21 15:54:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-21 15:54:00,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-21 15:54:00,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-21 15:54:00,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-21 15:54:00,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-21 15:54:00,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-21 15:54:00,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-21 15:54:00,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-21 15:54:00,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-21 15:54:00,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-21 15:54:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-21 15:54:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-21 15:54:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-21 15:54:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-21 15:54:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-21 15:54:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-21 15:54:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-21 15:54:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-21 15:54:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-21 15:54:00,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-21 15:54:00,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-21 15:54:00,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-21 15:54:00,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-21 15:54:00,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-21 15:54:00,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-21 15:54:00,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-21 15:54:00,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-21 15:54:00,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-21 15:54:00,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-21 15:54:00,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-21 15:54:00,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-21 15:54:00,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-21 15:54:00,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-21 15:54:00,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-21 15:54:00,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-21 15:54:00,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-21 15:54:00,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-21 15:54:00,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-21 15:54:00,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-21 15:54:00,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-21 15:54:00,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-21 15:54:00,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-21 15:54:00,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-21 15:54:00,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-21 15:54:00,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-21 15:54:00,233 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-21 15:54:00,233 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-21 15:54:00,233 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-21 15:54:01,023 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 942 ms to try 5000 different iterations. Found a solution that moves 4685 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17400068832859275. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04003614777696594, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0564891257177402, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5205555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04003614777696594, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16933333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23408333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-21 15:54:01,071 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv110175828:83 , srv1411337809:83 , srv1671021654:83 , srv1739441805:83 , srv1775272251:83 , srv233556544:83 , srv63284980:83 , srv781135591:83 , srv1086482623:84 , srv1224366248:84 , srv1538937595:84 , srv1642416006:84 , srv1663160707:84 , srv1711271568:84 , srv1801434977:84 , srv2043369321:84 , srv461706654:84 , srv520306952:84 , srv774618466:84 , srv77879529:84 , srv812365250:84 , srv851908634:84 , srv1149793524:85 , srv1278928537:85 , srv1385708941:85 , srv1453678693:85 , srv1473777661:85 , srv1528512911:85 , srv1588698834:85 , srv1588861858:85 , srv1702584796:85 , srv1877369653:85 , srv1925978285:85 , srv194804301:85 , srv344120215:85 , srv363414844:85 , srv440216289:85 , srv468397816:85 , srv503426113:85 , srv966947036:85 , srv1270265089:86 , srv1774729582:86 , srv1829016380:86 , srv1831220036:86 , srv1967703473:86 , srv2036127434:86 , srv2125702422:86 , srv2131382946:86 , srv458949034:86 , srv797328769:86 , srv1047805028:87 , srv1192136238:87 , srv1280026373:87 , srv1289739963:87 , srv1449219727:87 , srv1455900192:87 , srv1483194069:87 , srv1534882723:87 , srv1795778349:87 , srv1816747805:87 , srv1825550077:87 , srv1887568951:87 , srv1990690109:87 , srv1995193228:87 , srv2082294497:87 , srv315976577:87 , srv330917977:87 , srv433251492:87 , srv532058390:87 , srv681989867:87 , srv77838584:87 , srv781037534:87 , srv85632200:87 , srv987634221:87 , srv1020800256:88 , srv12890847:88 , srv1416367717:88 , srv1476014151:88 , srv1728042007:88 , srv1947066012:88 , srv2018046033:88 , srv207526673:88 , srv241109374:88 , srv924369892:88 , srv1133778216:89 , srv1626444925:89 , srv173649663:89 , srv342754561:89 , srv470602390:89 , srv888361000:89 , srv1096163443:90 , srv1268956503:90 , srv1396946349:90 , srv1918662982:90 , srv1509399884:91 , srv1870401905:92 , srv1962075914:93 , srv274993801:93 , srv872429626:93 , srv337444073:446 } 2023-02-21 15:54:01,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-21 15:54:01,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1829016380=46, srv2082294497=63, srv1133778216=5, srv468397816=79, srv1702584796=35, srv1774729582=40, srv1990690109=57, srv851908634=93, srv2018046033=59, srv363414844=74, srv1816747805=44, srv503426113=81, srv330917977=70, srv1671021654=34, srv1270265089=10, srv797328769=91, srv1483194069=24, srv2043369321=61, srv781135591=90, srv1509399884=25, srv77879529=88, srv872429626=95, srv1887568951=50, srv1925978285=52, srv77838584=87, srv85632200=94, srv2036127434=60, srv1289739963=14, srv1962075914=55, srv342754561=72, srv1528512911=26, srv1268956503=9, srv1453678693=20, srv1870401905=48, srv207526673=62, srv888361000=96, srv458949034=77, srv1396946349=16, srv1280026373=12, srv1096163443=3, srv1473777661=22, srv1825550077=45, srv1588698834=29, srv194804301=54, srv315976577=69, srv1047805028=1, srv337444073=71, srv774618466=86, srv440216289=76, srv532058390=83, srv966947036=98, srv1642416006=32, srv1967703473=56, srv233556544=66, srv1538937595=28, srv1801434977=43, srv1711271568=36, srv1278928537=11, srv1020800256=0, srv1224366248=8, srv1831220036=47, srv681989867=85, srv1663160707=33, srv1476014151=23, srv924369892=97, srv63284980=84, srv12890847=13, srv1416367717=18, srv1626444925=31, srv1385708941=15, srv1534882723=27, srv1728042007=37, srv1449219727=19, srv2131382946=65, srv1588861858=30, srv1149793524=6, srv173649663=38, srv1877369653=49, srv781037534=89, srv520306952=82, srv433251492=75, srv110175828=4, srv1192136238=7, srv2125702422=64, srv470602390=80, srv987634221=99, srv1995193228=58, srv1739441805=39, srv1086482623=2, srv1918662982=51, srv461706654=78, srv1947066012=53, srv241109374=67, srv1795778349=42, srv1411337809=17, srv1455900192=21, srv344120215=73, srv1775272251=41, srv812365250=92, srv274993801=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-21 15:54:01,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-21 15:54:01,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-21 15:54:01,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-21 15:54:01,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-21 15:54:01,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-21 15:54:01,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-21 15:54:01,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-21 15:54:01,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-21 15:54:01,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-21 15:54:01,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-21 15:54:01,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-21 15:54:01,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-21 15:54:01,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-21 15:54:01,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-21 15:54:01,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-21 15:54:01,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-21 15:54:01,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-21 15:54:01,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-21 15:54:01,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-21 15:54:01,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-21 15:54:01,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-21 15:54:01,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-21 15:54:01,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-21 15:54:01,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-21 15:54:01,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-21 15:54:01,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-21 15:54:01,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-21 15:54:01,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-21 15:54:01,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-21 15:54:01,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-21 15:54:01,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-21 15:54:01,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-21 15:54:01,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-21 15:54:01,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-21 15:54:01,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-21 15:54:01,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-21 15:54:01,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-21 15:54:01,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-21 15:54:01,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-21 15:54:01,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-21 15:54:01,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-21 15:54:01,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-21 15:54:01,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-21 15:54:01,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-21 15:54:01,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-21 15:54:01,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-21 15:54:01,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-21 15:54:01,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-21 15:54:01,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-21 15:54:01,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-21 15:54:01,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-21 15:54:01,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-21 15:54:01,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-21 15:54:01,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-21 15:54:01,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-21 15:54:01,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-21 15:54:01,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-21 15:54:01,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-21 15:54:01,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-21 15:54:01,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-21 15:54:01,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-21 15:54:01,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-21 15:54:01,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-21 15:54:01,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-21 15:54:01,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-21 15:54:01,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-21 15:54:01,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-21 15:54:01,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-21 15:54:01,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-21 15:54:01,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-21 15:54:01,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-21 15:54:01,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-21 15:54:01,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-21 15:54:01,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-21 15:54:01,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-21 15:54:01,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-21 15:54:01,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-21 15:54:01,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-21 15:54:01,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-21 15:54:01,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-21 15:54:01,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-21 15:54:01,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-21 15:54:01,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-21 15:54:01,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-21 15:54:01,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-21 15:54:01,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-21 15:54:01,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-21 15:54:01,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-21 15:54:01,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-21 15:54:01,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-21 15:54:01,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-21 15:54:01,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-21 15:54:01,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-21 15:54:01,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-21 15:54:01,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-21 15:54:01,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-21 15:54:01,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-21 15:54:01,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-21 15:54:01,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-21 15:54:01,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-21 15:54:01,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-21 15:54:01,244 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04003614777696594, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0564891257177402, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04003614777696594, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16933333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23408333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-21 15:54:01,244 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-21 15:54:01,245 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17396788325743592, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04003614777696594, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0564891257177402, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04003614777696594, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16933333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23408333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-21 15:54:01,529 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 399 ms to try 5000 different iterations. Found a solution that moves 2611 regions; Going from a computed imbalance of 0.17396788325743592 to a new imbalance of 0.01490614225724632. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02386937056623682); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03519067828820351, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2901111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02386937056623682); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-21 15:54:01,542 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1962075914:79 , srv872429626:79 , srv1268956503:80 , srv1728042007:81 , srv1870401905:81 , srv1918662982:81 , srv1476014151:82 , srv173649663:82 , srv1947066012:82 , srv207526673:82 , srv470602390:82 , srv924369892:82 , srv1133778216:83 , srv1509399884:83 , srv1626444925:83 , srv1795778349:83 , srv1831220036:83 , srv2018046033:83 , srv241109374:83 , srv433251492:83 , srv681989867:83 , srv781037534:83 , srv1020800256:84 , srv1096163443:84 , srv1289739963:84 , srv1825550077:84 , srv274993801:84 , srv532058390:84 , srv888361000:84 , srv1047805028:85 , srv1149793524:85 , srv1192136238:85 , srv1278928537:85 , srv1396946349:85 , srv1449219727:85 , srv1588698834:85 , srv1588861858:85 , srv1887568951:85 , srv1925978285:85 , srv194804301:85 , srv1990690109:85 , srv2082294497:85 , srv2131382946:85 , srv315976577:85 , srv342754561:85 , srv458949034:85 , srv468397816:85 , srv77838584:85 , srv1224366248:86 , srv1270265089:86 , srv1280026373:86 , srv12890847:86 , srv1385708941:86 , srv1411337809:86 , srv1483194069:86 , srv1528512911:86 , srv1702584796:86 , srv1711271568:86 , srv1739441805:86 , srv1774729582:86 , srv1829016380:86 , srv1877369653:86 , srv1967703473:86 , srv330917977:86 , srv77879529:86 , srv1416367717:87 , srv1455900192:87 , srv1473777661:87 , srv1534882723:87 , srv1538937595:87 , srv1642416006:87 , srv1801434977:87 , srv1995193228:87 , srv2043369321:87 , srv344120215:87 , srv440216289:87 , srv520306952:87 , srv774618466:87 , srv781135591:87 , srv797328769:87 , srv812365250:87 , srv85632200:87 , srv1086482623:88 , srv1453678693:88 , srv1663160707:88 , srv1816747805:88 , srv2036127434:88 , srv363414844:88 , srv503426113:88 , srv110175828:89 , srv1671021654:89 , srv233556544:89 , srv851908634:89 , srv966947036:89 , srv987634221:89 , srv2125702422:90 , srv63284980:90 , srv1775272251:91 , srv461706654:92 , srv337444073:540 } 2023-02-21 15:54:01,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-21 15:54:01,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1829016380=46, srv2082294497=63, srv1133778216=5, srv468397816=79, srv1702584796=35, srv1774729582=40, srv1990690109=57, srv851908634=93, srv2018046033=59, srv363414844=74, srv1816747805=44, srv503426113=81, srv330917977=70, srv1671021654=34, srv1270265089=10, srv797328769=91, srv1483194069=24, srv2043369321=61, srv781135591=90, srv1509399884=25, srv77879529=88, srv872429626=95, srv1887568951=50, srv1925978285=52, srv77838584=87, srv85632200=94, srv2036127434=60, srv1289739963=14, srv1962075914=55, srv342754561=72, srv1528512911=26, srv1268956503=9, srv1453678693=20, srv1870401905=48, srv207526673=62, srv888361000=96, srv458949034=77, srv1396946349=16, srv1280026373=12, srv1096163443=3, srv1473777661=22, srv1825550077=45, srv1588698834=29, srv194804301=54, srv315976577=69, srv1047805028=1, srv337444073=71, srv774618466=86, srv440216289=76, srv532058390=83, srv966947036=98, srv1642416006=32, srv1967703473=56, srv233556544=66, srv1538937595=28, srv1801434977=43, srv1711271568=36, srv1278928537=11, srv1020800256=0, srv1224366248=8, srv1831220036=47, srv681989867=85, srv1663160707=33, srv1476014151=23, srv924369892=97, srv63284980=84, srv12890847=13, srv1416367717=18, srv1626444925=31, srv1385708941=15, srv1534882723=27, srv1728042007=37, srv1449219727=19, srv2131382946=65, srv1588861858=30, srv1149793524=6, srv173649663=38, srv1877369653=49, srv781037534=89, srv520306952=82, srv433251492=75, srv110175828=4, srv1192136238=7, srv2125702422=64, srv470602390=80, srv987634221=99, srv1995193228=58, srv1739441805=39, srv1086482623=2, srv1918662982=51, srv461706654=78, srv1947066012=53, srv241109374=67, srv1795778349=42, srv1411337809=17, srv1455900192=21, srv344120215=73, srv1775272251=41, srv812365250=92, srv274993801=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-21 15:54:01,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-21 15:54:01,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-21 15:54:01,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-21 15:54:01,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-21 15:54:01,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-21 15:54:01,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-21 15:54:01,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-21 15:54:01,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-21 15:54:01,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-21 15:54:01,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-21 15:54:01,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-21 15:54:01,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-21 15:54:01,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-21 15:54:01,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-21 15:54:01,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-21 15:54:01,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-21 15:54:01,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-21 15:54:01,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-21 15:54:01,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-21 15:54:01,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-21 15:54:01,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-21 15:54:01,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-21 15:54:01,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-21 15:54:01,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-21 15:54:01,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-21 15:54:01,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-21 15:54:01,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-21 15:54:01,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-21 15:54:01,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-21 15:54:01,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-21 15:54:01,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-21 15:54:01,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-21 15:54:01,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-21 15:54:01,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-21 15:54:01,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-21 15:54:01,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-21 15:54:01,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-21 15:54:01,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-21 15:54:01,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-21 15:54:01,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-21 15:54:01,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-21 15:54:01,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-21 15:54:01,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-21 15:54:01,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-21 15:54:01,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-21 15:54:01,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-21 15:54:01,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-21 15:54:01,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-21 15:54:01,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-21 15:54:01,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-21 15:54:01,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-21 15:54:01,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-21 15:54:01,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-21 15:54:01,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-21 15:54:01,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-21 15:54:01,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-21 15:54:01,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-21 15:54:01,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-21 15:54:01,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-21 15:54:01,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-21 15:54:01,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-21 15:54:01,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-21 15:54:01,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-21 15:54:01,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-21 15:54:01,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-21 15:54:01,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-21 15:54:01,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-21 15:54:01,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-21 15:54:01,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-21 15:54:01,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-21 15:54:01,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-21 15:54:01,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-21 15:54:01,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-21 15:54:01,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-21 15:54:01,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-21 15:54:01,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-21 15:54:01,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-21 15:54:01,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-21 15:54:01,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-21 15:54:01,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-21 15:54:01,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-21 15:54:01,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-21 15:54:01,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-21 15:54:01,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-21 15:54:01,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-21 15:54:01,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-21 15:54:01,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-21 15:54:01,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-21 15:54:01,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-21 15:54:01,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-21 15:54:01,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-21 15:54:01,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-21 15:54:01,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-21 15:54:01,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-21 15:54:01,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-21 15:54:01,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-21 15:54:01,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-21 15:54:01,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-21 15:54:01,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-21 15:54:01,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-21 15:54:01,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-21 15:54:01,740 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02386937056623682); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03519067828820351, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02386937056623682); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-21 15:54:01,740 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-21 15:54:01,740 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014887859644484204, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02386937056623682); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03519067828820351, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02386937056623682); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-21 15:54:01,994 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 375 ms to try 5000 different iterations. Found a solution that moves 973 regions; Going from a computed imbalance of 0.014887859644484204 to a new imbalance of 7.982699530511963E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008122088723203193); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01213467266075249); MoveCostFunction : (multiplier=7.0, imbalance=0.10811111111111112, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008122088723203193); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-21 15:54:01,996 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1416367717:81 , srv2125702422:81 , srv274993801:81 , srv987634221:81 , srv1396946349:82 , srv1509399884:82 , srv1816747805:82 , srv461706654:82 , srv1096163443:83 , srv12890847:83 , srv1455900192:83 , srv1483194069:83 , srv1534882723:83 , srv1775272251:83 , srv1995193228:83 , srv851908634:83 , srv85632200:83 , srv1086482623:84 , srv1453678693:84 , srv1663160707:84 , srv2036127434:84 , srv330917977:84 , srv363414844:84 , srv503426113:84 , srv63284980:84 , srv797328769:84 , srv888361000:84 , srv966947036:84 , srv1020800256:85 , srv1047805028:85 , srv110175828:85 , srv1133778216:85 , srv1192136238:85 , srv1270265089:85 , srv1280026373:85 , srv1449219727:85 , srv1642416006:85 , srv1671021654:85 , srv1774729582:85 , srv1870401905:85 , srv1877369653:85 , srv1962075914:85 , srv1990690109:85 , srv2082294497:85 , srv233556544:85 , srv241109374:85 , srv315976577:85 , srv342754561:85 , srv344120215:85 , srv872429626:85 , srv1385708941:86 , srv1473777661:86 , srv1538937595:86 , srv1801434977:86 , srv1829016380:86 , srv1887568951:86 , srv1918662982:86 , srv1967703473:86 , srv2018046033:86 , srv2131382946:86 , srv458949034:86 , srv532058390:86 , srv77838584:86 , srv812365250:86 , srv1149793524:87 , srv1268956503:87 , srv1278928537:87 , srv1476014151:87 , srv1588698834:87 , srv1626444925:87 , srv1702584796:87 , srv1711271568:87 , srv1728042007:87 , srv173649663:87 , srv1825550077:87 , srv1947066012:87 , srv194804301:87 , srv2043369321:87 , srv207526673:87 , srv468397816:87 , srv470602390:87 , srv520306952:87 , srv774618466:87 , srv77879529:87 , srv781037534:87 , srv924369892:87 , srv1224366248:88 , srv1289739963:88 , srv1528512911:88 , srv1588861858:88 , srv1739441805:88 , srv1795778349:88 , srv1831220036:88 , srv1925978285:88 , srv433251492:88 , srv440216289:88 , srv681989867:88 , srv781135591:88 , srv1411337809:89 , srv337444073:544 } 2023-02-21 15:54:02,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-21 15:54:02,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1829016380=46, srv2082294497=63, srv1133778216=5, srv468397816=79, srv1702584796=35, srv1774729582=40, srv1990690109=57, srv851908634=93, srv2018046033=59, srv363414844=74, srv1816747805=44, srv503426113=81, srv330917977=70, srv1671021654=34, srv1270265089=10, srv797328769=91, srv1483194069=24, srv2043369321=61, srv781135591=90, srv1509399884=25, srv77879529=88, srv872429626=95, srv1887568951=50, srv1925978285=52, srv77838584=87, srv85632200=94, srv2036127434=60, srv1289739963=14, srv1962075914=55, srv342754561=72, srv1528512911=26, srv1268956503=9, srv1453678693=20, srv1870401905=48, srv207526673=62, srv888361000=96, srv458949034=77, srv1396946349=16, srv1280026373=12, srv1096163443=3, srv1473777661=22, srv1825550077=45, srv1588698834=29, srv194804301=54, srv315976577=69, srv1047805028=1, srv337444073=71, srv774618466=86, srv440216289=76, srv532058390=83, srv966947036=98, srv1642416006=32, srv1967703473=56, srv233556544=66, srv1538937595=28, srv1801434977=43, srv1711271568=36, srv1278928537=11, srv1020800256=0, srv1224366248=8, srv1831220036=47, srv681989867=85, srv1663160707=33, srv1476014151=23, srv924369892=97, srv63284980=84, srv12890847=13, srv1416367717=18, srv1626444925=31, srv1385708941=15, srv1534882723=27, srv1728042007=37, srv1449219727=19, srv2131382946=65, srv1588861858=30, srv1149793524=6, srv173649663=38, srv1877369653=49, srv781037534=89, srv520306952=82, srv433251492=75, srv110175828=4, srv1192136238=7, srv2125702422=64, srv470602390=80, srv987634221=99, srv1995193228=58, srv1739441805=39, srv1086482623=2, srv1918662982=51, srv461706654=78, srv1947066012=53, srv241109374=67, srv1795778349=42, srv1411337809=17, srv1455900192=21, srv344120215=73, srv1775272251=41, srv812365250=92, srv274993801=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-21 15:54:02,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-21 15:54:02,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-21 15:54:02,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-21 15:54:02,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-21 15:54:02,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-21 15:54:02,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-21 15:54:02,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-21 15:54:02,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-21 15:54:02,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-21 15:54:02,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-21 15:54:02,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-21 15:54:02,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-21 15:54:02,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-21 15:54:02,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-21 15:54:02,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-21 15:54:02,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-21 15:54:02,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-21 15:54:02,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-21 15:54:02,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-21 15:54:02,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-21 15:54:02,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-21 15:54:02,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-21 15:54:02,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-21 15:54:02,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-21 15:54:02,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-21 15:54:02,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-21 15:54:02,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-21 15:54:02,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-21 15:54:02,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-21 15:54:02,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-21 15:54:02,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-21 15:54:02,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-21 15:54:02,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-21 15:54:02,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-21 15:54:02,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-21 15:54:02,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-21 15:54:02,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-21 15:54:02,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-21 15:54:02,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-21 15:54:02,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-21 15:54:02,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-21 15:54:02,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-21 15:54:02,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-21 15:54:02,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-21 15:54:02,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-21 15:54:02,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-21 15:54:02,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-21 15:54:02,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-21 15:54:02,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-21 15:54:02,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-21 15:54:02,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-21 15:54:02,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-21 15:54:02,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-21 15:54:02,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-21 15:54:02,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-21 15:54:02,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-21 15:54:02,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-21 15:54:02,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-21 15:54:02,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-21 15:54:02,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-21 15:54:02,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-21 15:54:02,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-21 15:54:02,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-21 15:54:02,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-21 15:54:02,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-21 15:54:02,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-21 15:54:02,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-21 15:54:02,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-21 15:54:02,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-21 15:54:02,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-21 15:54:02,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-21 15:54:02,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-21 15:54:02,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-21 15:54:02,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-21 15:54:02,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-21 15:54:02,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-21 15:54:02,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-21 15:54:02,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-21 15:54:02,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-21 15:54:02,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-21 15:54:02,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-21 15:54:02,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-21 15:54:02,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-21 15:54:02,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-21 15:54:02,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-21 15:54:02,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-21 15:54:02,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-21 15:54:02,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-21 15:54:02,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-21 15:54:02,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-21 15:54:02,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-21 15:54:02,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-21 15:54:02,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-21 15:54:02,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-21 15:54:02,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-21 15:54:02,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-21 15:54:02,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-21 15:54:02,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-21 15:54:02,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-21 15:54:02,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-21 15:54:02,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-21 15:54:02,167 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008122088723203193); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01213467266075249); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008122088723203193); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-21 15:54:02,168 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-21 15:54:02,168 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.914568614320693E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008122088723203193); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01213467266075249); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008122088723203193); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-21 15:54:02,534 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 487 ms to try 5000 different iterations. Found a solution that moves 293 regions; Going from a computed imbalance of 7.914568614320693E-4 to a new imbalance of 5.024523486598299E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.03255555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-21 15:54:02,535 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv440216289:82 , srv1289739963:83 , srv1528512911:83 , srv1626444925:83 , srv1702584796:83 , srv1224366248:84 , srv1280026373:84 , srv1385708941:84 , srv1411337809:84 , srv1449219727:84 , srv1473777661:84 , srv1476014151:84 , srv1588861858:84 , srv173649663:84 , srv1739441805:84 , srv1795778349:84 , srv1825550077:84 , srv1829016380:84 , srv1887568951:84 , srv1918662982:84 , srv1925978285:84 , srv1967703473:84 , srv2036127434:84 , srv2131382946:84 , srv274993801:84 , srv342754561:84 , srv433251492:84 , srv461706654:84 , srv470602390:84 , srv520306952:84 , srv681989867:84 , srv774618466:84 , srv77838584:84 , srv781135591:84 , srv966947036:84 , srv1047805028:85 , srv1096163443:85 , srv110175828:85 , srv1133778216:85 , srv1149793524:85 , srv1192136238:85 , srv1270265089:85 , srv1278928537:85 , srv12890847:85 , srv1396946349:85 , srv1453678693:85 , srv1455900192:85 , srv1509399884:85 , srv1534882723:85 , srv1588698834:85 , srv1671021654:85 , srv1711271568:85 , srv1775272251:85 , srv1801434977:85 , srv1816747805:85 , srv1831220036:85 , srv1870401905:85 , srv1947066012:85 , srv194804301:85 , srv1962075914:85 , srv1990690109:85 , srv1995193228:85 , srv2018046033:85 , srv2043369321:85 , srv207526673:85 , srv2082294497:85 , srv2125702422:85 , srv233556544:85 , srv315976577:85 , srv330917977:85 , srv344120215:85 , srv363414844:85 , srv458949034:85 , srv468397816:85 , srv503426113:85 , srv532058390:85 , srv63284980:85 , srv77879529:85 , srv797328769:85 , srv812365250:85 , srv85632200:85 , srv872429626:85 , srv888361000:85 , srv924369892:85 , srv987634221:85 , srv1020800256:86 , srv1086482623:86 , srv1268956503:86 , srv1416367717:86 , srv1483194069:86 , srv1538937595:86 , srv1642416006:86 , srv1663160707:86 , srv1728042007:86 , srv1774729582:86 , srv1877369653:86 , srv241109374:86 , srv781037534:86 , srv851908634:86 , srv337444073:612 } 2023-02-21 15:54:02,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-21 15:54:02,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1829016380=46, srv2082294497=63, srv1133778216=5, srv468397816=79, srv1702584796=35, srv1774729582=40, srv1990690109=57, srv851908634=93, srv2018046033=59, srv363414844=74, srv1816747805=44, srv503426113=81, srv330917977=70, srv1671021654=34, srv1270265089=10, srv797328769=91, srv1483194069=24, srv2043369321=61, srv781135591=90, srv1509399884=25, srv77879529=88, srv872429626=95, srv1887568951=50, srv1925978285=52, srv77838584=87, srv85632200=94, srv2036127434=60, srv1289739963=14, srv1962075914=55, srv342754561=72, srv1528512911=26, srv1268956503=9, srv1453678693=20, srv1870401905=48, srv207526673=62, srv888361000=96, srv458949034=77, srv1396946349=16, srv1280026373=12, srv1096163443=3, srv1473777661=22, srv1825550077=45, srv1588698834=29, srv194804301=54, srv315976577=69, srv1047805028=1, srv337444073=71, srv774618466=86, srv440216289=76, srv532058390=83, srv966947036=98, srv1642416006=32, srv1967703473=56, srv233556544=66, srv1538937595=28, srv1801434977=43, srv1711271568=36, srv1278928537=11, srv1020800256=0, srv1224366248=8, srv1831220036=47, srv681989867=85, srv1663160707=33, srv1476014151=23, srv924369892=97, srv63284980=84, srv12890847=13, srv1416367717=18, srv1626444925=31, srv1385708941=15, srv1534882723=27, srv1728042007=37, srv1449219727=19, srv2131382946=65, srv1588861858=30, srv1149793524=6, srv173649663=38, srv1877369653=49, srv781037534=89, srv520306952=82, srv433251492=75, srv110175828=4, srv1192136238=7, srv2125702422=64, srv470602390=80, srv987634221=99, srv1995193228=58, srv1739441805=39, srv1086482623=2, srv1918662982=51, srv461706654=78, srv1947066012=53, srv241109374=67, srv1795778349=42, srv1411337809=17, srv1455900192=21, srv344120215=73, srv1775272251=41, srv812365250=92, srv274993801=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-21 15:54:02,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-21 15:54:02,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-21 15:54:02,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-21 15:54:02,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-21 15:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-21 15:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-21 15:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-21 15:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-21 15:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-21 15:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-21 15:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-21 15:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-21 15:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-21 15:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-21 15:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-21 15:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-21 15:54:02,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-21 15:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-21 15:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-21 15:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-21 15:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-21 15:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-21 15:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-21 15:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-21 15:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-21 15:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-21 15:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-21 15:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-21 15:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-21 15:54:02,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-21 15:54:02,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-21 15:54:02,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-21 15:54:02,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-21 15:54:02,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-21 15:54:02,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-21 15:54:02,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-21 15:54:02,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-21 15:54:02,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-21 15:54:02,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-21 15:54:02,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-21 15:54:02,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-21 15:54:02,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-21 15:54:02,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-21 15:54:02,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-21 15:54:02,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-21 15:54:02,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-21 15:54:02,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-21 15:54:02,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-21 15:54:02,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-21 15:54:02,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-21 15:54:02,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-21 15:54:02,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-21 15:54:02,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-21 15:54:02,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-21 15:54:02,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-21 15:54:02,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-21 15:54:02,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-21 15:54:02,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-21 15:54:02,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-21 15:54:02,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-21 15:54:02,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-21 15:54:02,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-21 15:54:02,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-21 15:54:02,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-21 15:54:02,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-21 15:54:02,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-21 15:54:02,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-21 15:54:02,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-21 15:54:02,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-21 15:54:02,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-21 15:54:02,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-21 15:54:02,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-21 15:54:02,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-21 15:54:02,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-21 15:54:02,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-21 15:54:02,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-21 15:54:02,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-21 15:54:02,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-21 15:54:02,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-21 15:54:02,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-21 15:54:02,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-21 15:54:02,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-21 15:54:02,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-21 15:54:02,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-21 15:54:02,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-21 15:54:02,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-21 15:54:02,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-21 15:54:02,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-21 15:54:02,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-21 15:54:02,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-21 15:54:02,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-21 15:54:02,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-21 15:54:02,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-21 15:54:02,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-21 15:54:02,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-21 15:54:02,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-21 15:54:02,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-21 15:54:02,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-21 15:54:02,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-21 15:54:02,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-21 15:54:02,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-21 15:54:02,682 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-21 15:54:02,682 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-02-21 15:54:02,682 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.819360501561893E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-02-21 15:54:02,912 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 334 ms to try 5000 different iterations. Found a solution that moves 32 regions; Going from a computed imbalance of 4.819360501561893E-5 to a new imbalance of 7.297308085447707E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0035555555555555557); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-21 15:54:02,913 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1020800256:83 , srv1268956503:83 , srv1538937595:83 , srv1774729582:83 , srv1870401905:83 , srv2043369321:83 , srv342754561:83 , srv781037534:83 , srv1047805028:84 , srv1086482623:84 , srv1096163443:84 , srv110175828:84 , srv1133778216:84 , srv1149793524:84 , srv1192136238:84 , srv1224366248:84 , srv1270265089:84 , srv1278928537:84 , srv1280026373:84 , srv12890847:84 , srv1289739963:84 , srv1396946349:84 , srv1411337809:84 , srv1416367717:84 , srv1453678693:84 , srv1455900192:84 , srv1473777661:84 , srv1483194069:84 , srv1528512911:84 , srv1534882723:84 , srv1588698834:84 , srv1588861858:84 , srv1626444925:84 , srv1642416006:84 , srv1663160707:84 , srv1671021654:84 , srv1711271568:84 , srv1728042007:84 , srv173649663:84 , srv1775272251:84 , srv1795778349:84 , srv1801434977:84 , srv1816747805:84 , srv1825550077:84 , srv1829016380:84 , srv1831220036:84 , srv1877369653:84 , srv1887568951:84 , srv1925978285:84 , srv1947066012:84 , srv194804301:84 , srv1962075914:84 , srv1967703473:84 , srv1990690109:84 , srv1995193228:84 , srv2018046033:84 , srv2036127434:84 , srv207526673:84 , srv2082294497:84 , srv2125702422:84 , srv233556544:84 , srv241109374:84 , srv274993801:84 , srv315976577:84 , srv330917977:84 , srv344120215:84 , srv363414844:84 , srv433251492:84 , srv440216289:84 , srv458949034:84 , srv461706654:84 , srv468397816:84 , srv470602390:84 , srv503426113:84 , srv520306952:84 , srv532058390:84 , srv63284980:84 , srv681989867:84 , srv774618466:84 , srv77838584:84 , srv77879529:84 , srv781135591:84 , srv797328769:84 , srv812365250:84 , srv851908634:84 , srv85632200:84 , srv872429626:84 , srv888361000:84 , srv924369892:84 , srv966947036:84 , srv987634221:84 , srv1385708941:85 , srv1449219727:85 , srv1476014151:85 , srv1509399884:85 , srv1702584796:85 , srv1739441805:85 , srv1918662982:85 , srv2131382946:85 , srv337444073:684 } 2023-02-21 15:54:02,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-21 15:54:02,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1829016380=46, srv2082294497=63, srv1133778216=5, srv468397816=79, srv1702584796=35, srv1774729582=40, srv1990690109=57, srv851908634=93, srv2018046033=59, srv363414844=74, srv1816747805=44, srv503426113=81, srv330917977=70, srv1671021654=34, srv1270265089=10, srv797328769=91, srv1483194069=24, srv2043369321=61, srv781135591=90, srv1509399884=25, srv77879529=88, srv872429626=95, srv1887568951=50, srv1925978285=52, srv77838584=87, srv85632200=94, srv2036127434=60, srv1289739963=14, srv1962075914=55, srv342754561=72, srv1528512911=26, srv1268956503=9, srv1453678693=20, srv1870401905=48, srv207526673=62, srv888361000=96, srv458949034=77, srv1396946349=16, srv1280026373=12, srv1096163443=3, srv1473777661=22, srv1825550077=45, srv1588698834=29, srv194804301=54, srv315976577=69, srv1047805028=1, srv337444073=71, srv774618466=86, srv440216289=76, srv532058390=83, srv966947036=98, srv1642416006=32, srv1967703473=56, srv233556544=66, srv1538937595=28, srv1801434977=43, srv1711271568=36, srv1278928537=11, srv1020800256=0, srv1224366248=8, srv1831220036=47, srv681989867=85, srv1663160707=33, srv1476014151=23, srv924369892=97, srv63284980=84, srv12890847=13, srv1416367717=18, srv1626444925=31, srv1385708941=15, srv1534882723=27, srv1728042007=37, srv1449219727=19, srv2131382946=65, srv1588861858=30, srv1149793524=6, srv173649663=38, srv1877369653=49, srv781037534=89, srv520306952=82, srv433251492=75, srv110175828=4, srv1192136238=7, srv2125702422=64, srv470602390=80, srv987634221=99, srv1995193228=58, srv1739441805=39, srv1086482623=2, srv1918662982=51, srv461706654=78, srv1947066012=53, srv241109374=67, srv1795778349=42, srv1411337809=17, srv1455900192=21, srv344120215=73, srv1775272251=41, srv812365250=92, srv274993801=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-21 15:54:02,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-21 15:54:02,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-21 15:54:02,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-21 15:54:02,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-21 15:54:02,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-21 15:54:02,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-21 15:54:02,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-21 15:54:02,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-21 15:54:02,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-21 15:54:02,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-21 15:54:02,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-21 15:54:02,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-21 15:54:02,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-21 15:54:02,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-21 15:54:02,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-21 15:54:02,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-21 15:54:02,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-21 15:54:02,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-21 15:54:02,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-21 15:54:02,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-21 15:54:02,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-21 15:54:02,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-21 15:54:02,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-21 15:54:02,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-21 15:54:02,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-21 15:54:02,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-21 15:54:02,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-21 15:54:02,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-21 15:54:02,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-21 15:54:02,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-21 15:54:02,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-02-21 15:54:02,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-02-21 15:54:02,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-02-21 15:54:02,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-02-21 15:54:02,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-02-21 15:54:02,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-02-21 15:54:02,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-02-21 15:54:02,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-02-21 15:54:02,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-02-21 15:54:02,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-02-21 15:54:02,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-02-21 15:54:02,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-02-21 15:54:02,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-02-21 15:54:02,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-02-21 15:54:02,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-02-21 15:54:02,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-02-21 15:54:02,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-02-21 15:54:02,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-02-21 15:54:02,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-02-21 15:54:02,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-02-21 15:54:02,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-02-21 15:54:02,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-02-21 15:54:02,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-02-21 15:54:02,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-02-21 15:54:02,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-02-21 15:54:02,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-02-21 15:54:02,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-02-21 15:54:02,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-02-21 15:54:02,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-02-21 15:54:02,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-02-21 15:54:02,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-02-21 15:54:02,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-02-21 15:54:02,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-02-21 15:54:02,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-02-21 15:54:02,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-02-21 15:54:02,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-02-21 15:54:02,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-02-21 15:54:02,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-02-21 15:54:02,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-02-21 15:54:02,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-02-21 15:54:02,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-02-21 15:54:02,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-02-21 15:54:02,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-02-21 15:54:02,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-02-21 15:54:02,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-02-21 15:54:02,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-02-21 15:54:02,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-02-21 15:54:02,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-02-21 15:54:02,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-02-21 15:54:02,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-02-21 15:54:02,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-02-21 15:54:02,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-02-21 15:54:02,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-02-21 15:54:02,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-02-21 15:54:02,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-02-21 15:54:02,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-02-21 15:54:02,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-02-21 15:54:02,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-02-21 15:54:02,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-02-21 15:54:02,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-02-21 15:54:02,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-02-21 15:54:02,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-02-21 15:54:02,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-02-21 15:54:02,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-02-21 15:54:02,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-02-21 15:54:02,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-02-21 15:54:02,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-02-21 15:54:02,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-02-21 15:54:02,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-02-21 15:54:02,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-02-21 15:54:02,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-02-21 15:54:03,071 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-21 15:54:03,072 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1020800256:83 , srv1268956503:83 , srv1538937595:83 , srv1774729582:83 , srv1870401905:83 , srv2043369321:83 , srv342754561:83 , srv781037534:83 , srv1047805028:84 , srv1086482623:84 , srv1096163443:84 , srv110175828:84 , srv1133778216:84 , srv1149793524:84 , srv1192136238:84 , srv1224366248:84 , srv1270265089:84 , srv1278928537:84 , srv1280026373:84 , srv12890847:84 , srv1289739963:84 , srv1396946349:84 , srv1411337809:84 , srv1416367717:84 , srv1453678693:84 , srv1455900192:84 , srv1473777661:84 , srv1483194069:84 , srv1528512911:84 , srv1534882723:84 , srv1588698834:84 , srv1588861858:84 , srv1626444925:84 , srv1642416006:84 , srv1663160707:84 , srv1671021654:84 , srv1711271568:84 , srv1728042007:84 , srv173649663:84 , srv1775272251:84 , srv1795778349:84 , srv1801434977:84 , srv1816747805:84 , srv1825550077:84 , srv1829016380:84 , srv1831220036:84 , srv1877369653:84 , srv1887568951:84 , srv1925978285:84 , srv1947066012:84 , srv194804301:84 , srv1962075914:84 , srv1967703473:84 , srv1990690109:84 , srv1995193228:84 , srv2018046033:84 , srv2036127434:84 , srv207526673:84 , srv2082294497:84 , srv2125702422:84 , srv233556544:84 , srv241109374:84 , srv274993801:84 , srv315976577:84 , srv330917977:84 , srv344120215:84 , srv363414844:84 , srv433251492:84 , srv440216289:84 , srv458949034:84 , srv461706654:84 , srv468397816:84 , srv470602390:84 , srv503426113:84 , srv520306952:84 , srv532058390:84 , srv63284980:84 , srv681989867:84 , srv774618466:84 , srv77838584:84 , srv77879529:84 , srv781135591:84 , srv797328769:84 , srv812365250:84 , srv851908634:84 , srv85632200:84 , srv872429626:84 , srv888361000:84 , srv924369892:84 , srv966947036:84 , srv987634221:84 , srv1385708941:85 , srv1449219727:85 , srv1476014151:85 , srv1509399884:85 , srv1702584796:85 , srv1739441805:85 , srv1918662982:85 , srv2131382946:85 , srv337444073:684 } 2023-02-21 15:54:03,098 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=336 (was 330) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=9251 (was 9769) 2023-02-21 15:54:03,106 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=336, ProcessCount=172, AvailableMemoryMB=9250 2023-02-21 15:54:03,107 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-02-21 15:54:03,107 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-02-21 15:54:03,118 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1102777378:84 , srv1113478459:84 , srv1116236552:84 , srv1166705936:84 , srv1194298862:84 , srv1197994974:84 , srv1225845757:84 , srv1447816234:84 , srv1599597470:84 , srv1748296097:84 , srv177107242:84 , srv1771507773:84 , srv1826867201:84 , srv1954542403:84 , srv1964954364:84 , srv2005760619:84 , srv214114573:84 , srv30346259:84 , srv400401322:84 , srv454228394:84 , srv495509993:84 , srv5097117:84 , srv519847317:84 , srv555871940:84 , srv76985135:84 , srv807310245:84 , srv848653783:84 , srv859736135:84 , srv88215319:84 , srv1457140853:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-02-21 15:54:03,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-21 15:54:03,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv555871940=24, srv1113478459=1, srv1102777378=0, srv88215319=29, srv1166705936=3, srv519847317=23, srv1771507773=12, srv1748296097=10, srv1964954364=15, srv1197994974=5, srv1447816234=7, srv30346259=18, srv1954542403=14, srv214114573=17, srv2005760619=16, srv1225845757=6, srv859736135=28, srv454228394=20, srv1826867201=13, srv495509993=21, srv76985135=25, srv1194298862=4, srv5097117=22, srv1116236552=2, srv1457140853=8, srv177107242=11, srv1599597470=9, srv848653783=27, srv400401322=19, srv807310245=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-21 15:54:03,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-21 15:54:03,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-21 15:54:03,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-21 15:54:03,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-21 15:54:03,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-21 15:54:03,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-21 15:54:03,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-21 15:54:03,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-21 15:54:03,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-21 15:54:03,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-21 15:54:03,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-21 15:54:03,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-21 15:54:03,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-21 15:54:03,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-21 15:54:03,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-21 15:54:03,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-21 15:54:03,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-21 15:54:03,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-21 15:54:03,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-21 15:54:03,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-21 15:54:03,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-21 15:54:03,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-21 15:54:03,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-21 15:54:03,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-21 15:54:03,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-21 15:54:03,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-21 15:54:03,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-21 15:54:03,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-21 15:54:03,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-21 15:54:03,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-21 15:54:03,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-21 15:54:03,155 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0613624572739835, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-21 15:54:03,155 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912386394363483, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0613624572739835, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-02-21 15:56:03,139 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6811529 different iterations. Found a solution that moves 1834 regions; Going from a computed imbalance of 0.9912386394363483 to a new imbalance of 4.280647492113412E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6792592592592592, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-21 15:56:03,145 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1102777378:90 , srv1113478459:90 , srv1116236552:90 , srv1166705936:90 , srv1194298862:90 , srv1197994974:90 , srv1225845757:90 , srv1447816234:90 , srv1457140853:90 , srv1599597470:90 , srv1748296097:90 , srv177107242:90 , srv1771507773:90 , srv1826867201:90 , srv1954542403:90 , srv1964954364:90 , srv2005760619:90 , srv214114573:90 , srv30346259:90 , srv400401322:90 , srv454228394:90 , srv495509993:90 , srv5097117:90 , srv519847317:90 , srv555871940:90 , srv76985135:90 , srv807310245:90 , srv848653783:90 , srv859736135:90 , srv88215319:90 } 2023-02-21 15:56:03,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-02-21 15:56:03,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv555871940=24, srv1113478459=1, srv1102777378=0, srv88215319=29, srv1166705936=3, srv519847317=23, srv1771507773=12, srv1748296097=10, srv1964954364=15, srv1197994974=5, srv1447816234=7, srv30346259=18, srv1954542403=14, srv214114573=17, srv2005760619=16, srv1225845757=6, srv859736135=28, srv454228394=20, srv1826867201=13, srv495509993=21, srv76985135=25, srv1194298862=4, srv5097117=22, srv1116236552=2, srv1457140853=8, srv177107242=11, srv1599597470=9, srv848653783=27, srv400401322=19, srv807310245=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-02-21 15:56:03,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-21 15:56:03,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-02-21 15:56:03,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-02-21 15:56:03,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-02-21 15:56:03,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-02-21 15:56:03,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-02-21 15:56:03,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-02-21 15:56:03,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-02-21 15:56:03,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-02-21 15:56:03,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-02-21 15:56:03,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-02-21 15:56:03,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-02-21 15:56:03,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-02-21 15:56:03,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-02-21 15:56:03,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-02-21 15:56:03,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-02-21 15:56:03,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-02-21 15:56:03,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-02-21 15:56:03,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-02-21 15:56:03,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-02-21 15:56:03,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-02-21 15:56:03,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-02-21 15:56:03,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-02-21 15:56:03,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-02-21 15:56:03,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-02-21 15:56:03,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-02-21 15:56:03,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-02-21 15:56:03,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-02-21 15:56:03,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-02-21 15:56:03,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-02-21 15:56:03,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-02-21 15:56:03,162 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-02-21 15:56:03,163 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1102777378:90 , srv1113478459:90 , srv1116236552:90 , srv1166705936:90 , srv1194298862:90 , srv1197994974:90 , srv1225845757:90 , srv1447816234:90 , srv1457140853:90 , srv1599597470:90 , srv1748296097:90 , srv177107242:90 , srv1771507773:90 , srv1826867201:90 , srv1954542403:90 , srv1964954364:90 , srv2005760619:90 , srv214114573:90 , srv30346259:90 , srv400401322:90 , srv454228394:90 , srv495509993:90 , srv5097117:90 , srv519847317:90 , srv555871940:90 , srv76985135:90 , srv807310245:90 , srv848653783:90 , srv859736135:90 , srv88215319:90 } 2023-02-21 15:56:03,175 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=202 (was 336), ProcessCount=172 (was 172), AvailableMemoryMB=9082 (was 9250)