2023-03-21 20:55:59,966 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-03-21 20:56:00,287 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-21 20:56:00,406 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-21 20:56:00,473 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-21 20:56:00,480 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-21 20:56:00,493 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=400, ProcessCount=169, AvailableMemoryMB=5531 2023-03-21 20:56:00,495 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-21 20:56:00,496 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-21 20:56:00,630 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1001778906:84 , srv1077388390:84 , srv1081082056:84 , srv108441417:84 , srv1087233103:84 , srv1088942398:84 , srv1105387966:84 , srv1108620097:84 , srv1182006748:84 , srv1192617116:84 , srv1202871834:84 , srv123004058:84 , srv126071741:84 , srv1270886375:84 , srv1275032583:84 , srv1306353617:84 , srv1319144530:84 , srv1339867530:84 , srv1351215177:84 , srv1360687901:84 , srv1386583627:84 , srv1408408133:84 , srv1412378579:84 , srv1416093058:84 , srv1461232249:84 , srv147415482:84 , srv1498708033:84 , srv1521201941:84 , srv1545301441:84 , srv154541253:84 , srv1554374377:84 , srv1568794639:84 , srv157880132:84 , srv1586939123:84 , srv1586965921:84 , srv1587482607:84 , srv1635752585:84 , srv1656384383:84 , srv1674570572:84 , srv1689237108:84 , srv169072416:84 , srv1690982431:84 , srv1693272845:84 , srv1722274375:84 , srv1768754370:84 , srv1775300018:84 , srv1812123274:84 , srv1815806672:84 , srv1816263491:84 , srv1864001275:84 , srv186588109:84 , srv1899725134:84 , srv1916436344:84 , srv1939202058:84 , srv1941829084:84 , srv1945549421:84 , srv1956388745:84 , srv1989673051:84 , srv2005363321:84 , srv2069773782:84 , srv2091633364:84 , srv2096524655:84 , srv212793416:84 , srv257344432:84 , srv294636633:84 , srv325199405:84 , srv334240439:84 , srv343516036:84 , srv357856650:84 , srv387904224:84 , srv438748140:84 , srv499847804:84 , srv546463780:84 , srv582616792:84 , srv612908936:84 , srv61574721:84 , srv646354126:84 , srv652580064:84 , srv654600250:84 , srv664856565:84 , srv670838440:84 , srv712921387:84 , srv726001733:84 , srv753661339:84 , srv781218479:84 , srv790699633:84 , srv820688942:84 , srv832498547:84 , srv870282474:84 , srv885649695:84 , srv88582170:84 , srv887489330:84 , srv897668234:84 , srv927239095:84 , srv93867881:84 , srv945471378:84 , srv952255498:84 , srv981501318:84 , srv990063474:84 , srv652836960:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-03-21 20:56:00,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-21 20:56:00,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1105387966=6, srv981501318=98, srv325199405=65, srv726001733=83, srv1864001275=49, srv61574721=75, srv1088942398=5, srv820688942=87, srv257344432=63, srv897668234=93, srv1545301441=28, srv2096524655=61, srv1275032583=14, srv126071741=12, srv1108620097=7, srv2091633364=60, srv1674570572=38, srv1339867530=17, srv1812123274=46, srv1416093058=23, srv1412378579=22, srv157880132=32, srv1816263491=48, srv612908936=74, srv546463780=72, srv1916436344=52, srv438748140=70, srv1077388390=1, srv1939202058=53, srv887489330=92, srv646354126=76, srv154541253=29, srv88582170=91, srv1775300018=45, srv1693272845=42, srv781218479=85, srv1899725134=51, srv1081082056=2, srv664856565=80, srv1689237108=39, srv147415482=25, srv1386583627=20, srv654600250=79, srv952255498=97, srv387904224=69, srv1568794639=31, srv169072416=40, srv1815806672=47, srv343516036=67, srv1690982431=41, srv334240439=66, srv1722274375=43, srv186588109=50, srv1461232249=24, srv2069773782=59, srv1087233103=4, srv294636633=64, srv1521201941=27, srv1202871834=10, srv1941829084=54, srv499847804=71, srv1408408133=21, srv790699633=86, srv945471378=96, srv885649695=90, srv1554374377=30, srv1498708033=26, srv1586965921=34, srv870282474=89, srv1656384383=37, srv1945549421=55, srv212793416=62, srv1360687901=19, srv123004058=11, srv927239095=94, srv1001778906=0, srv2005363321=58, srv93867881=95, srv582616792=73, srv1182006748=8, srv1319144530=16, srv832498547=88, srv1768754370=44, srv652836960=78, srv1351215177=18, srv1587482607=35, srv1586939123=33, srv670838440=81, srv1270886375=13, srv1635752585=36, srv357856650=68, srv753661339=84, srv712921387=82, srv1989673051=57, srv1306353617=15, srv108441417=3, srv1956388745=56, srv1192617116=9, srv652580064=77, srv990063474=99} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-21 20:56:00,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-21 20:56:00,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-21 20:56:00,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-21 20:56:00,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-21 20:56:00,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-21 20:56:00,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-21 20:56:00,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-21 20:56:00,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-21 20:56:00,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-21 20:56:00,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-21 20:56:00,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-21 20:56:00,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-21 20:56:00,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-21 20:56:00,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-21 20:56:00,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-21 20:56:00,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-21 20:56:00,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-21 20:56:00,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-21 20:56:00,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-21 20:56:00,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-21 20:56:00,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-21 20:56:00,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-21 20:56:00,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-21 20:56:00,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-21 20:56:00,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-21 20:56:00,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-21 20:56:00,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-21 20:56:00,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-21 20:56:00,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-21 20:56:00,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-21 20:56:00,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-21 20:56:00,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-21 20:56:00,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-21 20:56:00,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-21 20:56:00,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-21 20:56:00,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-21 20:56:00,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-21 20:56:00,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-21 20:56:00,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-21 20:56:00,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-21 20:56:00,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-21 20:56:00,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-21 20:56:00,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-21 20:56:00,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-21 20:56:00,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-21 20:56:00,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-21 20:56:00,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-21 20:56:00,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-21 20:56:00,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-21 20:56:00,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-21 20:56:00,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-21 20:56:00,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-21 20:56:00,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-21 20:56:00,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-21 20:56:00,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-21 20:56:00,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-21 20:56:00,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-21 20:56:00,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-21 20:56:00,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-21 20:56:00,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-21 20:56:00,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-21 20:56:00,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-21 20:56:00,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-21 20:56:00,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-21 20:56:00,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-21 20:56:00,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-21 20:56:00,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-21 20:56:00,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-21 20:56:00,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-21 20:56:00,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-21 20:56:00,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-21 20:56:00,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-21 20:56:00,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-21 20:56:00,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-21 20:56:00,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-21 20:56:00,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-21 20:56:00,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-21 20:56:00,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-21 20:56:00,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-21 20:56:00,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-21 20:56:00,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-21 20:56:00,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-21 20:56:00,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-21 20:56:00,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-21 20:56:00,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-21 20:56:00,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-21 20:56:00,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-21 20:56:00,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-21 20:56:00,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-21 20:56:00,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-21 20:56:00,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-21 20:56:00,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-21 20:56:00,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-21 20:56:00,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-21 20:56:00,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-21 20:56:00,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-21 20:56:00,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-21 20:56:00,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-21 20:56:00,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-21 20:56:00,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-21 20:56:00,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-21 20:56:00,990 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-21 20:56:00,991 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-21 20:56:00,991 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-21 20:56:01,643 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 878 ms to try 5000 different iterations. Found a solution that moves 4726 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17667359853262174. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04148148148148148, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06034320100863538, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5251111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04148148148148148, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17166666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24016666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-21 20:56:01,665 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1077388390:82 , srv1306353617:83 , srv1689237108:83 , srv1864001275:83 , srv1939202058:83 , srv438748140:83 , srv646354126:83 , srv88582170:83 , srv1088942398:84 , srv1192617116:84 , srv147415482:84 , srv1568794639:84 , srv1656384383:84 , srv1674570572:84 , srv1722274375:84 , srv212793416:84 , srv343516036:84 , srv753661339:84 , srv93867881:84 , srv952255498:84 , srv1081082056:85 , srv108441417:85 , srv1182006748:85 , srv1202871834:85 , srv1408408133:85 , srv1412378579:85 , srv1545301441:85 , srv154541253:85 , srv157880132:85 , srv1586939123:85 , srv1586965921:85 , srv1587482607:85 , srv1635752585:85 , srv1815806672:85 , srv325199405:85 , srv334240439:85 , srv357856650:85 , srv61574721:85 , srv670838440:85 , srv712921387:85 , srv123004058:86 , srv1339867530:86 , srv1351215177:86 , srv1386583627:86 , srv1498708033:86 , srv1521201941:86 , srv1693272845:86 , srv1768754370:86 , srv1812123274:86 , srv1956388745:86 , srv387904224:86 , srv499847804:86 , srv612908936:86 , srv654600250:86 , srv832498547:86 , srv870282474:86 , srv981501318:86 , srv990063474:86 , srv1001778906:87 , srv1416093058:87 , srv1554374377:87 , srv169072416:87 , srv1899725134:87 , srv1916436344:87 , srv1941829084:87 , srv1945549421:87 , srv1989673051:87 , srv2005363321:87 , srv2069773782:87 , srv790699633:87 , srv820688942:87 , srv887489330:87 , srv897668234:87 , srv927239095:87 , srv1087233103:88 , srv1108620097:88 , srv126071741:88 , srv1360687901:88 , srv1816263491:88 , srv2096524655:88 , srv294636633:88 , srv546463780:88 , srv781218479:88 , srv1275032583:89 , srv1319144530:89 , srv1690982431:89 , srv2091633364:89 , srv257344432:89 , srv652580064:89 , srv726001733:89 , srv1105387966:90 , srv1270886375:90 , srv1461232249:90 , srv582616792:90 , srv664856565:90 , srv945471378:90 , srv885649695:91 , srv1775300018:92 , srv186588109:92 , srv652836960:459 } 2023-03-21 20:56:01,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-21 20:56:01,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1105387966=6, srv981501318=98, srv325199405=65, srv726001733=83, srv1864001275=49, srv61574721=75, srv1088942398=5, srv820688942=87, srv257344432=63, srv897668234=93, srv1545301441=28, srv2096524655=61, srv1275032583=14, srv126071741=12, srv1108620097=7, srv2091633364=60, srv1674570572=38, srv1339867530=17, srv1812123274=46, srv1416093058=23, srv1412378579=22, srv157880132=32, srv1816263491=48, srv612908936=74, srv546463780=72, srv1916436344=52, srv438748140=70, srv1077388390=1, srv1939202058=53, srv887489330=92, srv646354126=76, srv154541253=29, srv88582170=91, srv1775300018=45, srv1693272845=42, srv781218479=85, srv1899725134=51, srv1081082056=2, srv664856565=80, srv1689237108=39, srv147415482=25, srv1386583627=20, srv654600250=79, srv952255498=97, srv387904224=69, srv1568794639=31, srv169072416=40, srv1815806672=47, srv343516036=67, srv1690982431=41, srv334240439=66, srv1722274375=43, srv186588109=50, srv1461232249=24, srv2069773782=59, srv1087233103=4, srv294636633=64, srv1521201941=27, srv1202871834=10, srv1941829084=54, srv499847804=71, srv1408408133=21, srv790699633=86, srv945471378=96, srv885649695=90, srv1554374377=30, srv1498708033=26, srv1586965921=34, srv870282474=89, srv1656384383=37, srv1945549421=55, srv212793416=62, srv1360687901=19, srv123004058=11, srv927239095=94, srv1001778906=0, srv2005363321=58, srv93867881=95, srv582616792=73, srv1182006748=8, srv1319144530=16, srv832498547=88, srv1768754370=44, srv652836960=78, srv1351215177=18, srv1587482607=35, srv1586939123=33, srv670838440=81, srv1270886375=13, srv1635752585=36, srv357856650=68, srv753661339=84, srv712921387=82, srv1989673051=57, srv1306353617=15, srv108441417=3, srv1956388745=56, srv1192617116=9, srv652580064=77, srv990063474=99} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-21 20:56:01,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-21 20:56:01,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-21 20:56:01,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-21 20:56:01,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-21 20:56:01,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-21 20:56:01,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-21 20:56:01,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-21 20:56:01,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-21 20:56:01,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-21 20:56:01,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-21 20:56:01,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-21 20:56:01,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-21 20:56:01,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-21 20:56:01,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-21 20:56:01,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-21 20:56:01,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-21 20:56:01,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-21 20:56:01,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-21 20:56:01,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-21 20:56:01,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-21 20:56:01,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-21 20:56:01,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-21 20:56:01,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-21 20:56:01,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-21 20:56:01,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-21 20:56:01,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-21 20:56:01,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-21 20:56:01,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-21 20:56:01,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-21 20:56:01,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-21 20:56:01,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-21 20:56:01,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-21 20:56:01,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-21 20:56:01,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-21 20:56:01,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-21 20:56:01,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-21 20:56:01,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-21 20:56:01,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-21 20:56:01,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-21 20:56:01,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-21 20:56:01,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-21 20:56:01,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-21 20:56:01,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-21 20:56:01,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-21 20:56:01,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-21 20:56:01,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-21 20:56:01,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-21 20:56:01,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-21 20:56:01,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-21 20:56:01,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-21 20:56:01,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-21 20:56:01,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-21 20:56:01,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-21 20:56:01,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-21 20:56:01,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-21 20:56:01,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-21 20:56:01,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-21 20:56:01,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-21 20:56:01,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-21 20:56:01,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-21 20:56:01,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-21 20:56:01,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-21 20:56:01,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-21 20:56:01,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-21 20:56:01,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-21 20:56:01,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-21 20:56:01,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-21 20:56:01,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-21 20:56:01,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-21 20:56:01,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-21 20:56:01,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-21 20:56:01,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-21 20:56:01,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-21 20:56:01,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-21 20:56:01,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-21 20:56:01,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-21 20:56:01,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-21 20:56:01,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-21 20:56:01,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-21 20:56:01,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-21 20:56:01,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-21 20:56:01,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-21 20:56:01,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-21 20:56:01,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-21 20:56:01,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-21 20:56:01,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-21 20:56:01,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-21 20:56:01,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-21 20:56:01,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-21 20:56:01,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-21 20:56:01,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-21 20:56:01,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-21 20:56:01,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-21 20:56:01,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-21 20:56:01,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-21 20:56:01,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-21 20:56:01,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-21 20:56:01,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-21 20:56:01,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-21 20:56:01,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-21 20:56:01,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-21 20:56:01,913 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04148148148148148, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06034320100863538, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04148148148148148, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17166666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24016666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-21 20:56:01,913 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-21 20:56:01,913 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17664050637332881, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04148148148148148, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06034320100863538, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04148148148148148, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17166666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24016666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-21 20:56:02,228 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 513 ms to try 5000 different iterations. Found a solution that moves 2577 regions; Going from a computed imbalance of 0.17664050637332881 to a new imbalance of 0.01596342396920241. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021714169917564863); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034095659975731454, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28633333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021714169917564863); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-21 20:56:02,240 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1105387966:81 , srv1270886375:81 , srv1690982431:81 , srv186588109:81 , srv945471378:81 , srv1087233103:82 , srv1275032583:82 , srv1775300018:82 , srv664856565:82 , srv885649695:82 , srv1108620097:83 , srv126071741:83 , srv1319144530:83 , srv1816263491:83 , srv2096524655:83 , srv546463780:83 , srv582616792:83 , srv897668234:83 , srv1360687901:84 , srv1461232249:84 , srv1554374377:84 , srv169072416:84 , srv1768754370:84 , srv2005363321:84 , srv2091633364:84 , srv726001733:84 , srv820688942:84 , srv1001778906:85 , srv1386583627:85 , srv1635752585:85 , srv1956388745:85 , srv1989673051:85 , srv294636633:85 , srv357856650:85 , srv387904224:85 , srv652580064:85 , srv887489330:85 , srv927239095:85 , srv1081082056:86 , srv108441417:86 , srv1182006748:86 , srv1339867530:86 , srv1408408133:86 , srv1498708033:86 , srv1545301441:86 , srv157880132:86 , srv1656384383:86 , srv1693272845:86 , srv1899725134:86 , srv1916436344:86 , srv1941829084:86 , srv1945549421:86 , srv257344432:86 , srv343516036:86 , srv712921387:86 , srv781218479:86 , srv790699633:86 , srv870282474:86 , srv990063474:86 , srv123004058:87 , srv1306353617:87 , srv1351215177:87 , srv1412378579:87 , srv147415482:87 , srv154541253:87 , srv1568794639:87 , srv1674570572:87 , srv1815806672:87 , srv2069773782:87 , srv212793416:87 , srv325199405:87 , srv438748140:87 , srv499847804:87 , srv612908936:87 , srv654600250:87 , srv670838440:87 , srv832498547:87 , srv952255498:87 , srv1192617116:88 , srv1202871834:88 , srv1416093058:88 , srv1586965921:88 , srv1587482607:88 , srv1689237108:88 , srv1722274375:88 , srv334240439:88 , srv646354126:88 , srv753661339:88 , srv88582170:88 , srv981501318:88 , srv1521201941:89 , srv1586939123:89 , srv1812123274:89 , srv1864001275:89 , srv1939202058:89 , srv1077388390:90 , srv1088942398:90 , srv61574721:91 , srv93867881:91 , srv652836960:508 } 2023-03-21 20:56:02,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-21 20:56:02,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1105387966=6, srv981501318=98, srv325199405=65, srv726001733=83, srv1864001275=49, srv61574721=75, srv1088942398=5, srv820688942=87, srv257344432=63, srv897668234=93, srv1545301441=28, srv2096524655=61, srv1275032583=14, srv126071741=12, srv1108620097=7, srv2091633364=60, srv1674570572=38, srv1339867530=17, srv1812123274=46, srv1416093058=23, srv1412378579=22, srv157880132=32, srv1816263491=48, srv612908936=74, srv546463780=72, srv1916436344=52, srv438748140=70, srv1077388390=1, srv1939202058=53, srv887489330=92, srv646354126=76, srv154541253=29, srv88582170=91, srv1775300018=45, srv1693272845=42, srv781218479=85, srv1899725134=51, srv1081082056=2, srv664856565=80, srv1689237108=39, srv147415482=25, srv1386583627=20, srv654600250=79, srv952255498=97, srv387904224=69, srv1568794639=31, srv169072416=40, srv1815806672=47, srv343516036=67, srv1690982431=41, srv334240439=66, srv1722274375=43, srv186588109=50, srv1461232249=24, srv2069773782=59, srv1087233103=4, srv294636633=64, srv1521201941=27, srv1202871834=10, srv1941829084=54, srv499847804=71, srv1408408133=21, srv790699633=86, srv945471378=96, srv885649695=90, srv1554374377=30, srv1498708033=26, srv1586965921=34, srv870282474=89, srv1656384383=37, srv1945549421=55, srv212793416=62, srv1360687901=19, srv123004058=11, srv927239095=94, srv1001778906=0, srv2005363321=58, srv93867881=95, srv582616792=73, srv1182006748=8, srv1319144530=16, srv832498547=88, srv1768754370=44, srv652836960=78, srv1351215177=18, srv1587482607=35, srv1586939123=33, srv670838440=81, srv1270886375=13, srv1635752585=36, srv357856650=68, srv753661339=84, srv712921387=82, srv1989673051=57, srv1306353617=15, srv108441417=3, srv1956388745=56, srv1192617116=9, srv652580064=77, srv990063474=99} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-21 20:56:02,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-21 20:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-21 20:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-21 20:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-21 20:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-21 20:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-21 20:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-21 20:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-21 20:56:02,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-21 20:56:02,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-21 20:56:02,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-21 20:56:02,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-21 20:56:02,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-21 20:56:02,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-21 20:56:02,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-21 20:56:02,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-21 20:56:02,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-21 20:56:02,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-21 20:56:02,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-21 20:56:02,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-21 20:56:02,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-21 20:56:02,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-21 20:56:02,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-21 20:56:02,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-21 20:56:02,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-21 20:56:02,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-21 20:56:02,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-21 20:56:02,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-21 20:56:02,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-21 20:56:02,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-21 20:56:02,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-21 20:56:02,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-21 20:56:02,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-21 20:56:02,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-21 20:56:02,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-21 20:56:02,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-21 20:56:02,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-21 20:56:02,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-21 20:56:02,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-21 20:56:02,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-21 20:56:02,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-21 20:56:02,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-21 20:56:02,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-21 20:56:02,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-21 20:56:02,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-21 20:56:02,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-21 20:56:02,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-21 20:56:02,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-21 20:56:02,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-21 20:56:02,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-21 20:56:02,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-21 20:56:02,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-21 20:56:02,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-21 20:56:02,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-21 20:56:02,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-21 20:56:02,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-21 20:56:02,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-21 20:56:02,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-21 20:56:02,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-21 20:56:02,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-21 20:56:02,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-21 20:56:02,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-21 20:56:02,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-21 20:56:02,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-21 20:56:02,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-21 20:56:02,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-21 20:56:02,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-21 20:56:02,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-21 20:56:02,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-21 20:56:02,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-21 20:56:02,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-21 20:56:02,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-21 20:56:02,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-21 20:56:02,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-21 20:56:02,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-21 20:56:02,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-21 20:56:02,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-21 20:56:02,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-21 20:56:02,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-21 20:56:02,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-21 20:56:02,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-21 20:56:02,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-21 20:56:02,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-21 20:56:02,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-21 20:56:02,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-21 20:56:02,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-21 20:56:02,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-21 20:56:02,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-21 20:56:02,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-21 20:56:02,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-21 20:56:02,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-21 20:56:02,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-21 20:56:02,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-21 20:56:02,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-21 20:56:02,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-21 20:56:02,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-21 20:56:02,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-21 20:56:02,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-21 20:56:02,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-21 20:56:02,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-21 20:56:02,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-21 20:56:02,451 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021714169917564863); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034095659975731454, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021714169917564863); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-21 20:56:02,451 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-21 20:56:02,452 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01594537942952873, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021714169917564863); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034095659975731454, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021714169917564863); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04675, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-21 20:56:02,693 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 384 ms to try 5000 different iterations. Found a solution that moves 1017 regions; Going from a computed imbalance of 0.01594537942952873 to a new imbalance of 7.118080212338181E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00610013298189536); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008355190191882916); MoveCostFunction : (multiplier=7.0, imbalance=0.113, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00610013298189536); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-21 20:56:02,696 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv61574721:82 , srv1088942398:83 , srv1416093058:83 , srv1586939123:83 , srv1812123274:83 , srv1945549421:83 , srv257344432:83 , srv781218479:83 , srv981501318:83 , srv1202871834:84 , srv123004058:84 , srv1351215177:84 , srv1461232249:84 , srv1521201941:84 , srv1586965921:84 , srv1775300018:84 , srv186588109:84 , srv1916436344:84 , srv1941829084:84 , srv2069773782:84 , srv334240439:84 , srv612908936:84 , srv652580064:84 , srv885649695:84 , srv93867881:84 , srv1001778906:85 , srv1077388390:85 , srv1081082056:85 , srv1412378579:85 , srv1498708033:85 , srv154541253:85 , srv1587482607:85 , srv1693272845:85 , srv1864001275:85 , srv1899725134:85 , srv1939202058:85 , srv1989673051:85 , srv2091633364:85 , srv294636633:85 , srv499847804:85 , srv582616792:85 , srv654600250:85 , srv670838440:85 , srv726001733:85 , srv753661339:85 , srv790699633:85 , srv832498547:85 , srv870282474:85 , srv887489330:85 , srv927239095:85 , srv108441417:86 , srv1105387966:86 , srv1182006748:86 , srv1192617116:86 , srv126071741:86 , srv1275032583:86 , srv1319144530:86 , srv1339867530:86 , srv1360687901:86 , srv1408408133:86 , srv147415482:86 , srv1545301441:86 , srv1554374377:86 , srv157880132:86 , srv1674570572:86 , srv1722274375:86 , srv1815806672:86 , srv1816263491:86 , srv1956388745:86 , srv2005363321:86 , srv212793416:86 , srv325199405:86 , srv387904224:86 , srv546463780:86 , srv646354126:86 , srv945471378:86 , srv952255498:86 , srv990063474:86 , srv1087233103:87 , srv1108620097:87 , srv1270886375:87 , srv1386583627:87 , srv1568794639:87 , srv1635752585:87 , srv1689237108:87 , srv169072416:87 , srv1690982431:87 , srv1768754370:87 , srv2096524655:87 , srv357856650:87 , srv438748140:87 , srv712921387:87 , srv820688942:87 , srv88582170:87 , srv1306353617:88 , srv1656384383:88 , srv343516036:88 , srv664856565:88 , srv897668234:88 , srv652836960:545 } 2023-03-21 20:56:02,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-21 20:56:02,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1105387966=6, srv981501318=98, srv325199405=65, srv726001733=83, srv1864001275=49, srv61574721=75, srv1088942398=5, srv820688942=87, srv257344432=63, srv897668234=93, srv1545301441=28, srv2096524655=61, srv1275032583=14, srv126071741=12, srv1108620097=7, srv2091633364=60, srv1674570572=38, srv1339867530=17, srv1812123274=46, srv1416093058=23, srv1412378579=22, srv157880132=32, srv1816263491=48, srv612908936=74, srv546463780=72, srv1916436344=52, srv438748140=70, srv1077388390=1, srv1939202058=53, srv887489330=92, srv646354126=76, srv154541253=29, srv88582170=91, srv1775300018=45, srv1693272845=42, srv781218479=85, srv1899725134=51, srv1081082056=2, srv664856565=80, srv1689237108=39, srv147415482=25, srv1386583627=20, srv654600250=79, srv952255498=97, srv387904224=69, srv1568794639=31, srv169072416=40, srv1815806672=47, srv343516036=67, srv1690982431=41, srv334240439=66, srv1722274375=43, srv186588109=50, srv1461232249=24, srv2069773782=59, srv1087233103=4, srv294636633=64, srv1521201941=27, srv1202871834=10, srv1941829084=54, srv499847804=71, srv1408408133=21, srv790699633=86, srv945471378=96, srv885649695=90, srv1554374377=30, srv1498708033=26, srv1586965921=34, srv870282474=89, srv1656384383=37, srv1945549421=55, srv212793416=62, srv1360687901=19, srv123004058=11, srv927239095=94, srv1001778906=0, srv2005363321=58, srv93867881=95, srv582616792=73, srv1182006748=8, srv1319144530=16, srv832498547=88, srv1768754370=44, srv652836960=78, srv1351215177=18, srv1587482607=35, srv1586939123=33, srv670838440=81, srv1270886375=13, srv1635752585=36, srv357856650=68, srv753661339=84, srv712921387=82, srv1989673051=57, srv1306353617=15, srv108441417=3, srv1956388745=56, srv1192617116=9, srv652580064=77, srv990063474=99} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-21 20:56:02,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-21 20:56:02,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-21 20:56:02,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-21 20:56:02,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-21 20:56:02,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-21 20:56:02,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-21 20:56:02,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-21 20:56:02,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-21 20:56:02,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-21 20:56:02,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-21 20:56:02,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-21 20:56:02,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-21 20:56:02,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-21 20:56:02,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-21 20:56:02,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-21 20:56:02,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-21 20:56:02,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-21 20:56:02,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-21 20:56:02,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-21 20:56:02,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-21 20:56:02,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-21 20:56:02,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-21 20:56:02,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-21 20:56:02,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-21 20:56:02,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-21 20:56:02,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-21 20:56:02,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-21 20:56:02,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-21 20:56:02,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-21 20:56:02,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-21 20:56:02,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-21 20:56:02,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-21 20:56:02,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-21 20:56:02,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-21 20:56:02,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-21 20:56:02,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-21 20:56:02,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-21 20:56:02,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-21 20:56:02,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-21 20:56:02,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-21 20:56:02,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-21 20:56:02,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-21 20:56:02,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-21 20:56:02,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-21 20:56:02,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-21 20:56:02,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-21 20:56:02,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-21 20:56:02,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-21 20:56:02,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-21 20:56:02,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-21 20:56:02,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-21 20:56:02,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-21 20:56:02,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-21 20:56:02,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-21 20:56:02,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-21 20:56:02,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-21 20:56:02,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-21 20:56:02,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-21 20:56:02,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-21 20:56:02,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-21 20:56:02,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-21 20:56:02,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-21 20:56:02,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-21 20:56:02,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-21 20:56:02,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-21 20:56:02,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-21 20:56:02,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-21 20:56:02,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-21 20:56:02,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-21 20:56:02,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-21 20:56:02,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-21 20:56:02,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-21 20:56:02,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-21 20:56:02,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-21 20:56:02,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-21 20:56:02,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-21 20:56:02,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-21 20:56:02,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-21 20:56:02,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-21 20:56:02,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-21 20:56:02,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-21 20:56:02,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-21 20:56:02,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-21 20:56:02,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-21 20:56:02,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-21 20:56:02,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-21 20:56:02,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-21 20:56:02,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-21 20:56:02,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-21 20:56:02,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-21 20:56:02,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-21 20:56:02,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-21 20:56:02,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-21 20:56:02,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-21 20:56:02,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-21 20:56:02,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-21 20:56:02,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-21 20:56:02,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-21 20:56:02,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-21 20:56:02,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-21 20:56:02,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-21 20:56:02,828 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00610013298189536); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008355190191882916); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00610013298189536); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-21 20:56:02,828 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-21 20:56:02,828 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.046868350296534E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00610013298189536); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008355190191882916); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00610013298189536); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-21 20:56:03,140 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 408 ms to try 5000 different iterations. Found a solution that moves 266 regions; Going from a computed imbalance of 7.046868350296534E-4 to a new imbalance of 4.3319283717601865E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.029555555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-21 20:56:03,141 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv664856565:82 , srv1270886375:83 , srv1521201941:83 , srv93867881:83 , srv1105387966:84 , srv1108620097:84 , srv1192617116:84 , srv1275032583:84 , srv1306353617:84 , srv1319144530:84 , srv1339867530:84 , srv1351215177:84 , srv1360687901:84 , srv1386583627:84 , srv1416093058:84 , srv1568794639:84 , srv1587482607:84 , srv1656384383:84 , srv1689237108:84 , srv169072416:84 , srv1693272845:84 , srv1722274375:84 , srv1775300018:84 , srv1812123274:84 , srv1815806672:84 , srv186588109:84 , srv1899725134:84 , srv2005363321:84 , srv2069773782:84 , srv2091633364:84 , srv2096524655:84 , srv257344432:84 , srv294636633:84 , srv325199405:84 , srv343516036:84 , srv438748140:84 , srv499847804:84 , srv652580064:84 , srv654600250:84 , srv712921387:84 , srv726001733:84 , srv790699633:84 , srv832498547:84 , srv88582170:84 , srv897668234:84 , srv1001778906:85 , srv1077388390:85 , srv108441417:85 , srv1087233103:85 , srv1088942398:85 , srv1182006748:85 , srv1202871834:85 , srv1408408133:85 , srv1412378579:85 , srv1461232249:85 , srv147415482:85 , srv1498708033:85 , srv1545301441:85 , srv154541253:85 , srv1554374377:85 , srv157880132:85 , srv1586939123:85 , srv1586965921:85 , srv1635752585:85 , srv1674570572:85 , srv1690982431:85 , srv1768754370:85 , srv1816263491:85 , srv1864001275:85 , srv1916436344:85 , srv1939202058:85 , srv1941829084:85 , srv1956388745:85 , srv1989673051:85 , srv212793416:85 , srv334240439:85 , srv357856650:85 , srv387904224:85 , srv546463780:85 , srv582616792:85 , srv612908936:85 , srv61574721:85 , srv646354126:85 , srv670838440:85 , srv753661339:85 , srv781218479:85 , srv820688942:85 , srv870282474:85 , srv885649695:85 , srv887489330:85 , srv927239095:85 , srv945471378:85 , srv952255498:85 , srv981501318:85 , srv990063474:85 , srv1081082056:86 , srv123004058:86 , srv126071741:86 , srv1945549421:87 , srv652836960:630 } 2023-03-21 20:56:03,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-21 20:56:03,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1105387966=6, srv981501318=98, srv325199405=65, srv726001733=83, srv1864001275=49, srv61574721=75, srv1088942398=5, srv820688942=87, srv257344432=63, srv897668234=93, srv1545301441=28, srv2096524655=61, srv1275032583=14, srv126071741=12, srv1108620097=7, srv2091633364=60, srv1674570572=38, srv1339867530=17, srv1812123274=46, srv1416093058=23, srv1412378579=22, srv157880132=32, srv1816263491=48, srv612908936=74, srv546463780=72, srv1916436344=52, srv438748140=70, srv1077388390=1, srv1939202058=53, srv887489330=92, srv646354126=76, srv154541253=29, srv88582170=91, srv1775300018=45, srv1693272845=42, srv781218479=85, srv1899725134=51, srv1081082056=2, srv664856565=80, srv1689237108=39, srv147415482=25, srv1386583627=20, srv654600250=79, srv952255498=97, srv387904224=69, srv1568794639=31, srv169072416=40, srv1815806672=47, srv343516036=67, srv1690982431=41, srv334240439=66, srv1722274375=43, srv186588109=50, srv1461232249=24, srv2069773782=59, srv1087233103=4, srv294636633=64, srv1521201941=27, srv1202871834=10, srv1941829084=54, srv499847804=71, srv1408408133=21, srv790699633=86, srv945471378=96, srv885649695=90, srv1554374377=30, srv1498708033=26, srv1586965921=34, srv870282474=89, srv1656384383=37, srv1945549421=55, srv212793416=62, srv1360687901=19, srv123004058=11, srv927239095=94, srv1001778906=0, srv2005363321=58, srv93867881=95, srv582616792=73, srv1182006748=8, srv1319144530=16, srv832498547=88, srv1768754370=44, srv652836960=78, srv1351215177=18, srv1587482607=35, srv1586939123=33, srv670838440=81, srv1270886375=13, srv1635752585=36, srv357856650=68, srv753661339=84, srv712921387=82, srv1989673051=57, srv1306353617=15, srv108441417=3, srv1956388745=56, srv1192617116=9, srv652580064=77, srv990063474=99} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-21 20:56:03,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-21 20:56:03,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-21 20:56:03,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-21 20:56:03,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-21 20:56:03,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-21 20:56:03,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-21 20:56:03,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-21 20:56:03,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-21 20:56:03,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-21 20:56:03,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-21 20:56:03,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-21 20:56:03,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-21 20:56:03,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-21 20:56:03,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-21 20:56:03,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-21 20:56:03,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-21 20:56:03,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-21 20:56:03,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-21 20:56:03,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-21 20:56:03,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-21 20:56:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-21 20:56:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-21 20:56:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-21 20:56:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-21 20:56:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-21 20:56:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-21 20:56:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-21 20:56:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-21 20:56:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-21 20:56:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-21 20:56:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-21 20:56:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-21 20:56:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-21 20:56:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-21 20:56:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-21 20:56:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-21 20:56:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-21 20:56:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-21 20:56:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-21 20:56:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-21 20:56:03,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-21 20:56:03,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-21 20:56:03,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-21 20:56:03,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-21 20:56:03,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-21 20:56:03,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-21 20:56:03,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-21 20:56:03,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-21 20:56:03,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-21 20:56:03,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-21 20:56:03,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-21 20:56:03,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-21 20:56:03,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-21 20:56:03,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-21 20:56:03,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-21 20:56:03,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-21 20:56:03,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-21 20:56:03,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-21 20:56:03,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-21 20:56:03,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-21 20:56:03,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-21 20:56:03,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-21 20:56:03,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-21 20:56:03,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-21 20:56:03,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-21 20:56:03,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-21 20:56:03,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-21 20:56:03,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-21 20:56:03,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-21 20:56:03,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-21 20:56:03,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-21 20:56:03,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-21 20:56:03,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-21 20:56:03,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-21 20:56:03,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-21 20:56:03,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-21 20:56:03,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-21 20:56:03,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-21 20:56:03,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-21 20:56:03,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-21 20:56:03,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-21 20:56:03,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-21 20:56:03,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-21 20:56:03,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-21 20:56:03,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-21 20:56:03,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-21 20:56:03,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-21 20:56:03,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-21 20:56:03,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-21 20:56:03,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-21 20:56:03,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-21 20:56:03,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-21 20:56:03,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-21 20:56:03,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-21 20:56:03,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-21 20:56:03,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-21 20:56:03,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-21 20:56:03,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-21 20:56:03,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-21 20:56:03,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-21 20:56:03,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-21 20:56:03,263 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-21 20:56:03,263 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-21 20:56:03,264 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.1456711908056334E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-21 20:56:03,461 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 292 ms to try 5000 different iterations. Found a solution that moves 30 regions; Going from a computed imbalance of 4.1456711908056334E-5 to a new imbalance of 7.899486290149148E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0033333333333333335); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-21 20:56:03,461 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv123004058:83 , srv126071741:83 , srv1461232249:83 , srv1945549421:83 , srv582616792:83 , srv61574721:83 , srv820688942:83 , srv990063474:83 , srv1001778906:84 , srv1077388390:84 , srv1081082056:84 , srv108441417:84 , srv1087233103:84 , srv1088942398:84 , srv1108620097:84 , srv1182006748:84 , srv1192617116:84 , srv1202871834:84 , srv1306353617:84 , srv1319144530:84 , srv1339867530:84 , srv1360687901:84 , srv1386583627:84 , srv1408408133:84 , srv1412378579:84 , srv1416093058:84 , srv147415482:84 , srv1498708033:84 , srv1521201941:84 , srv1545301441:84 , srv154541253:84 , srv1554374377:84 , srv1568794639:84 , srv157880132:84 , srv1586939123:84 , srv1586965921:84 , srv1587482607:84 , srv1635752585:84 , srv1656384383:84 , srv1674570572:84 , srv1689237108:84 , srv169072416:84 , srv1690982431:84 , srv1722274375:84 , srv1768754370:84 , srv1775300018:84 , srv1812123274:84 , srv1815806672:84 , srv1816263491:84 , srv1864001275:84 , srv1899725134:84 , srv1916436344:84 , srv1939202058:84 , srv1941829084:84 , srv1956388745:84 , srv1989673051:84 , srv2069773782:84 , srv2091633364:84 , srv2096524655:84 , srv212793416:84 , srv257344432:84 , srv294636633:84 , srv325199405:84 , srv334240439:84 , srv343516036:84 , srv357856650:84 , srv387904224:84 , srv499847804:84 , srv546463780:84 , srv612908936:84 , srv646354126:84 , srv652580064:84 , srv654600250:84 , srv664856565:84 , srv670838440:84 , srv712921387:84 , srv726001733:84 , srv753661339:84 , srv781218479:84 , srv790699633:84 , srv832498547:84 , srv870282474:84 , srv885649695:84 , srv88582170:84 , srv887489330:84 , srv897668234:84 , srv927239095:84 , srv93867881:84 , srv945471378:84 , srv952255498:84 , srv981501318:84 , srv1105387966:85 , srv1270886375:85 , srv1275032583:85 , srv1351215177:85 , srv1693272845:85 , srv186588109:85 , srv2005363321:85 , srv438748140:85 , srv652836960:684 } 2023-03-21 20:56:03,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-21 20:56:03,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1105387966=6, srv981501318=98, srv325199405=65, srv726001733=83, srv1864001275=49, srv61574721=75, srv1088942398=5, srv820688942=87, srv257344432=63, srv897668234=93, srv1545301441=28, srv2096524655=61, srv1275032583=14, srv126071741=12, srv1108620097=7, srv2091633364=60, srv1674570572=38, srv1339867530=17, srv1812123274=46, srv1416093058=23, srv1412378579=22, srv157880132=32, srv1816263491=48, srv612908936=74, srv546463780=72, srv1916436344=52, srv438748140=70, srv1077388390=1, srv1939202058=53, srv887489330=92, srv646354126=76, srv154541253=29, srv88582170=91, srv1775300018=45, srv1693272845=42, srv781218479=85, srv1899725134=51, srv1081082056=2, srv664856565=80, srv1689237108=39, srv147415482=25, srv1386583627=20, srv654600250=79, srv952255498=97, srv387904224=69, srv1568794639=31, srv169072416=40, srv1815806672=47, srv343516036=67, srv1690982431=41, srv334240439=66, srv1722274375=43, srv186588109=50, srv1461232249=24, srv2069773782=59, srv1087233103=4, srv294636633=64, srv1521201941=27, srv1202871834=10, srv1941829084=54, srv499847804=71, srv1408408133=21, srv790699633=86, srv945471378=96, srv885649695=90, srv1554374377=30, srv1498708033=26, srv1586965921=34, srv870282474=89, srv1656384383=37, srv1945549421=55, srv212793416=62, srv1360687901=19, srv123004058=11, srv927239095=94, srv1001778906=0, srv2005363321=58, srv93867881=95, srv582616792=73, srv1182006748=8, srv1319144530=16, srv832498547=88, srv1768754370=44, srv652836960=78, srv1351215177=18, srv1587482607=35, srv1586939123=33, srv670838440=81, srv1270886375=13, srv1635752585=36, srv357856650=68, srv753661339=84, srv712921387=82, srv1989673051=57, srv1306353617=15, srv108441417=3, srv1956388745=56, srv1192617116=9, srv652580064=77, srv990063474=99} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-21 20:56:03,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-21 20:56:03,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-21 20:56:03,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-21 20:56:03,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-21 20:56:03,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-21 20:56:03,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-21 20:56:03,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-21 20:56:03,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-21 20:56:03,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-21 20:56:03,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-21 20:56:03,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-21 20:56:03,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-21 20:56:03,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-21 20:56:03,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-21 20:56:03,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-21 20:56:03,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-21 20:56:03,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-21 20:56:03,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-21 20:56:03,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-21 20:56:03,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-21 20:56:03,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-21 20:56:03,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-21 20:56:03,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-21 20:56:03,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-21 20:56:03,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-21 20:56:03,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-21 20:56:03,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-21 20:56:03,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-21 20:56:03,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-21 20:56:03,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-21 20:56:03,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-21 20:56:03,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-21 20:56:03,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-21 20:56:03,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-21 20:56:03,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-21 20:56:03,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-21 20:56:03,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-21 20:56:03,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-21 20:56:03,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-21 20:56:03,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-21 20:56:03,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-21 20:56:03,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-21 20:56:03,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-21 20:56:03,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-21 20:56:03,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-21 20:56:03,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-21 20:56:03,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-21 20:56:03,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-21 20:56:03,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-21 20:56:03,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-21 20:56:03,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-21 20:56:03,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-21 20:56:03,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-21 20:56:03,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-21 20:56:03,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-21 20:56:03,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-21 20:56:03,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-21 20:56:03,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-21 20:56:03,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-21 20:56:03,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-21 20:56:03,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-21 20:56:03,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-21 20:56:03,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-21 20:56:03,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-21 20:56:03,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-21 20:56:03,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-21 20:56:03,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-21 20:56:03,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-21 20:56:03,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-21 20:56:03,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-21 20:56:03,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-21 20:56:03,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-21 20:56:03,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-21 20:56:03,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-21 20:56:03,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-21 20:56:03,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-21 20:56:03,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-21 20:56:03,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-21 20:56:03,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-21 20:56:03,587 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.689421800350782E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-21 20:56:03,588 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv123004058:83 , srv126071741:83 , srv1461232249:83 , srv1945549421:83 , srv582616792:83 , srv61574721:83 , srv820688942:83 , srv990063474:83 , srv1001778906:84 , srv1077388390:84 , srv1081082056:84 , srv108441417:84 , srv1087233103:84 , srv1088942398:84 , srv1108620097:84 , srv1182006748:84 , srv1192617116:84 , srv1202871834:84 , srv1306353617:84 , srv1319144530:84 , srv1339867530:84 , srv1360687901:84 , srv1386583627:84 , srv1408408133:84 , srv1412378579:84 , srv1416093058:84 , srv147415482:84 , srv1498708033:84 , srv1521201941:84 , srv1545301441:84 , srv154541253:84 , srv1554374377:84 , srv1568794639:84 , srv157880132:84 , srv1586939123:84 , srv1586965921:84 , srv1587482607:84 , srv1635752585:84 , srv1656384383:84 , srv1674570572:84 , srv1689237108:84 , srv169072416:84 , srv1690982431:84 , srv1722274375:84 , srv1768754370:84 , srv1775300018:84 , srv1812123274:84 , srv1815806672:84 , srv1816263491:84 , srv1864001275:84 , srv1899725134:84 , srv1916436344:84 , srv1939202058:84 , srv1941829084:84 , srv1956388745:84 , srv1989673051:84 , srv2069773782:84 , srv2091633364:84 , srv2096524655:84 , srv212793416:84 , srv257344432:84 , srv294636633:84 , srv325199405:84 , srv334240439:84 , srv343516036:84 , srv357856650:84 , srv387904224:84 , srv499847804:84 , srv546463780:84 , srv612908936:84 , srv646354126:84 , srv652580064:84 , srv654600250:84 , srv664856565:84 , srv670838440:84 , srv712921387:84 , srv726001733:84 , srv753661339:84 , srv781218479:84 , srv790699633:84 , srv832498547:84 , srv870282474:84 , srv885649695:84 , srv88582170:84 , srv887489330:84 , srv897668234:84 , srv927239095:84 , srv93867881:84 , srv945471378:84 , srv952255498:84 , srv981501318:84 , srv1105387966:85 , srv1270886375:85 , srv1275032583:85 , srv1351215177:85 , srv1693272845:85 , srv186588109:85 , srv2005363321:85 , srv438748140:85 , srv652836960:684 } 2023-03-21 20:56:03,612 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=400 (was 400), ProcessCount=169 (was 169), AvailableMemoryMB=5048 (was 5531) 2023-03-21 20:56:03,620 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=400, ProcessCount=169, AvailableMemoryMB=5047 2023-03-21 20:56:03,620 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-21 20:56:03,621 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-03-21 20:56:03,629 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1106707100:84 , srv1280731164:84 , srv1388541550:84 , srv1424187088:84 , srv1449190665:84 , srv1558782223:84 , srv1730325115:84 , srv1734495711:84 , srv1791233831:84 , srv1912258955:84 , srv1919549487:84 , srv1952630059:84 , srv1963140922:84 , srv1969950446:84 , srv2071915587:84 , srv20724452:84 , srv2106059868:84 , srv264589845:84 , srv274777835:84 , srv363077307:84 , srv50565127:84 , srv534895226:84 , srv538193456:84 , srv542305556:84 , srv633155056:84 , srv704821176:84 , srv854157491:84 , srv862650769:84 , srv991089072:84 , srv1524067947:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-03-21 20:56:03,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-21 20:56:03,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv363077307=20, srv704821176=26, srv1952630059=12, srv1963140922=13, srv50565127=21, srv1730325115=7, srv2071915587=15, srv538193456=23, srv633155056=25, srv542305556=24, srv264589845=18, srv1524067947=5, srv1388541550=2, srv1449190665=4, srv854157491=27, srv274777835=19, srv1791233831=9, srv2106059868=17, srv1280731164=1, srv1912258955=10, srv1969950446=14, srv1919549487=11, srv862650769=28, srv534895226=22, srv20724452=16, srv1734495711=8, srv1106707100=0, srv991089072=29, srv1424187088=3, srv1558782223=6} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-21 20:56:03,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-21 20:56:03,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-21 20:56:03,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-21 20:56:03,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-21 20:56:03,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-21 20:56:03,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-21 20:56:03,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-21 20:56:03,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-21 20:56:03,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-03-21 20:56:03,656 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0904590279912576, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-21 20:56:03,656 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912478076707721, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0904590279912576, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-03-21 20:58:03,643 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6428125 different iterations. Found a solution that moves 1854 regions; Going from a computed imbalance of 0.9912478076707721 to a new imbalance of 4.3273284898463826E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6866666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-21 20:58:03,649 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1106707100:90 , srv1280731164:90 , srv1388541550:90 , srv1424187088:90 , srv1449190665:90 , srv1524067947:90 , srv1558782223:90 , srv1730325115:90 , srv1734495711:90 , srv1791233831:90 , srv1912258955:90 , srv1919549487:90 , srv1952630059:90 , srv1963140922:90 , srv1969950446:90 , srv2071915587:90 , srv20724452:90 , srv2106059868:90 , srv264589845:90 , srv274777835:90 , srv363077307:90 , srv50565127:90 , srv534895226:90 , srv538193456:90 , srv542305556:90 , srv633155056:90 , srv704821176:90 , srv854157491:90 , srv862650769:90 , srv991089072:90 } 2023-03-21 20:58:03,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-21 20:58:03,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv363077307=20, srv704821176=26, srv1952630059=12, srv1963140922=13, srv50565127=21, srv1730325115=7, srv2071915587=15, srv538193456=23, srv633155056=25, srv542305556=24, srv264589845=18, srv1524067947=5, srv1388541550=2, srv1449190665=4, srv854157491=27, srv274777835=19, srv1791233831=9, srv2106059868=17, srv1280731164=1, srv1912258955=10, srv1969950446=14, srv1919549487=11, srv862650769=28, srv534895226=22, srv20724452=16, srv1734495711=8, srv1106707100=0, srv991089072=29, srv1424187088=3, srv1558782223=6} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-21 20:58:03,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-21 20:58:03,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-21 20:58:03,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-21 20:58:03,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-21 20:58:03,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-21 20:58:03,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-21 20:58:03,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-21 20:58:03,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-03-21 20:58:03,666 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-21 20:58:03,666 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1106707100:90 , srv1280731164:90 , srv1388541550:90 , srv1424187088:90 , srv1449190665:90 , srv1524067947:90 , srv1558782223:90 , srv1730325115:90 , srv1734495711:90 , srv1791233831:90 , srv1912258955:90 , srv1919549487:90 , srv1952630059:90 , srv1963140922:90 , srv1969950446:90 , srv2071915587:90 , srv20724452:90 , srv2106059868:90 , srv264589845:90 , srv274777835:90 , srv363077307:90 , srv50565127:90 , srv534895226:90 , srv538193456:90 , srv542305556:90 , srv633155056:90 , srv704821176:90 , srv854157491:90 , srv862650769:90 , srv991089072:90 } 2023-03-21 20:58:03,679 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=194 (was 400), ProcessCount=168 (was 169), AvailableMemoryMB=5076 (was 5047) - AvailableMemoryMB LEAK? -