2023-07-18 03:53:12,929 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-07-18 03:53:13,260 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-18 03:53:13,374 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-18 03:53:13,442 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-18 03:53:13,452 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-18 03:53:13,469 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=328, ProcessCount=173, AvailableMemoryMB=7540 2023-07-18 03:53:13,472 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-18 03:53:13,472 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-18 03:53:13,614 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1050393381:84 , srv1057915999:84 , srv1066963939:84 , srv1081534190:84 , srv1115508933:84 , srv1119820446:84 , srv11280536:84 , srv1135013446:84 , srv1142056850:84 , srv1160458373:84 , srv1164448715:84 , srv1171046368:84 , srv1188261602:84 , srv1195808758:84 , srv120670372:84 , srv1209007601:84 , srv1228742511:84 , srv1240648072:84 , srv1246529318:84 , srv1249168840:84 , srv1253302287:84 , srv1275206049:84 , srv129885965:84 , srv1393238407:84 , srv1399720109:84 , srv1400622292:84 , srv1418354654:84 , srv1433137073:84 , srv1531884867:84 , srv1543691009:84 , srv1561600695:84 , srv1576444007:84 , srv1592742384:84 , srv1601420677:84 , srv1620622266:84 , srv1631883185:84 , srv1639036808:84 , srv1648235227:84 , srv167395310:84 , srv1684478712:84 , srv1741457984:84 , srv1778344114:84 , srv1811421904:84 , srv1814284943:84 , srv184152351:84 , srv1853788461:84 , srv1855687723:84 , srv1880127238:84 , srv1886333806:84 , srv1897759541:84 , srv1911569027:84 , srv1937862200:84 , srv1948055539:84 , srv1954651771:84 , srv1970686859:84 , srv1979796798:84 , srv2036783496:84 , srv2046423020:84 , srv207447048:84 , srv2108169847:84 , srv2110105774:84 , srv2119585016:84 , srv2124769815:84 , srv2143154442:84 , srv258608178:84 , srv270864186:84 , srv273548185:84 , srv300485011:84 , srv317426992:84 , srv349780047:84 , srv350024984:84 , srv365752328:84 , srv369349487:84 , srv394960129:84 , srv444742682:84 , srv451371961:84 , srv453476374:84 , srv486278932:84 , srv486768250:84 , srv551129837:84 , srv555058365:84 , srv572057717:84 , srv594072499:84 , srv660780393:84 , srv677179524:84 , srv714993098:84 , srv715025365:84 , srv77085764:84 , srv791961024:84 , srv79873209:84 , srv809595277:84 , srv878394429:84 , srv881856447:84 , srv88886804:84 , srv893291821:84 , srv93092426:84 , srv951576667:84 , srv991349757:84 , srv992370368:84 , srv633727769:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-07-18 03:53:13,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-18 03:53:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv93092426=96, srv677179524=85, srv1188261602=12, srv1811421904=42, srv2119585016=61, srv350024984=70, srv129885965=22, srv878394429=92, srv317426992=68, srv2108169847=59, srv951576667=97, srv992370368=99, srv2046423020=57, srv633727769=83, srv1050393381=0, srv11280536=6, srv453476374=76, srv486768250=78, srv1979796798=55, srv486278932=77, srv893291821=95, srv1171046368=11, srv1209007601=15, srv2036783496=56, srv120670372=14, srv369349487=72, srv1880127238=47, srv451371961=75, srv1531884867=28, srv1601420677=33, srv1057915999=1, srv1561600695=30, srv1249168840=19, srv207447048=58, srv1954651771=53, srv1886333806=48, srv273548185=66, srv1246529318=18, srv1066963939=2, srv77085764=88, srv1275206049=21, srv1160458373=9, srv1911569027=50, srv1778344114=41, srv270864186=65, srv88886804=94, srv258608178=64, srv167395310=38, srv1648235227=37, srv1081534190=3, srv1684478712=39, srv1853788461=45, srv2110105774=60, srv660780393=84, srv1592742384=32, srv1814284943=43, srv444742682=74, srv1897759541=49, srv2124769815=62, srv715025365=87, srv1142056850=8, srv1418354654=26, srv1631883185=35, srv1135013446=7, srv1937862200=51, srv1253302287=20, srv572057717=81, srv1970686859=54, srv1119820446=5, srv1399720109=24, srv991349757=98, srv1240648072=17, srv1576444007=31, srv1639036808=36, srv881856447=93, srv1433137073=27, srv394960129=73, srv1115508933=4, srv1741457984=40, srv551129837=79, srv791961024=89, srv79873209=90, srv714993098=86, srv1543691009=29, srv809595277=91, srv1620622266=34, srv1400622292=25, srv2143154442=63, srv1948055539=52, srv555058365=80, srv349780047=69, srv1855687723=46, srv365752328=71, srv1195808758=13, srv1228742511=16, srv184152351=44, srv300485011=67, srv1164448715=10, srv1393238407=23, srv594072499=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-18 03:53:13,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 03:53:13,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-18 03:53:13,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-18 03:53:13,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-18 03:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-18 03:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-18 03:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-18 03:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-18 03:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-18 03:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-18 03:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-18 03:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-18 03:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-18 03:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-18 03:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-18 03:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-18 03:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-18 03:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-18 03:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-18 03:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-18 03:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-18 03:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-18 03:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-18 03:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-18 03:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-18 03:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-18 03:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-18 03:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-18 03:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-18 03:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-18 03:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-18 03:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-18 03:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-18 03:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-18 03:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-18 03:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-18 03:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-18 03:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-18 03:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-18 03:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-18 03:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-18 03:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-18 03:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-18 03:53:13,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-18 03:53:13,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-18 03:53:13,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-18 03:53:13,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-18 03:53:13,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-18 03:53:13,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-18 03:53:13,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-18 03:53:13,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-18 03:53:13,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-18 03:53:13,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-18 03:53:13,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-18 03:53:13,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-18 03:53:13,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-18 03:53:13,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-18 03:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-18 03:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-18 03:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-18 03:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-18 03:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-18 03:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-18 03:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-18 03:53:13,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-18 03:53:13,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-18 03:53:13,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-18 03:53:13,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-18 03:53:13,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-18 03:53:13,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-18 03:53:13,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-18 03:53:13,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-18 03:53:13,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-18 03:53:13,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-18 03:53:13,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-18 03:53:13,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-18 03:53:13,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-18 03:53:13,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-18 03:53:13,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-18 03:53:13,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-18 03:53:13,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-18 03:53:13,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-18 03:53:13,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-18 03:53:13,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-18 03:53:13,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-18 03:53:13,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-18 03:53:13,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-18 03:53:13,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-18 03:53:13,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-18 03:53:13,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-18 03:53:13,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-18 03:53:13,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-18 03:53:13,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-18 03:53:13,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-18 03:53:13,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-18 03:53:13,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-18 03:53:13,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-18 03:53:13,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-18 03:53:13,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-18 03:53:13,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-18 03:53:13,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-18 03:53:14,003 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 03:53:14,003 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-18 03:53:14,003 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-18 03:53:14,419 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 648 ms to try 5000 different iterations. Found a solution that moves 4722 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1746817573588702. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040398316092815724, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05401292608314097, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5246666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040398316092815724, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16966666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23841666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 03:53:14,444 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv572057717:81 , srv1119820446:83 , srv1400622292:83 , srv1601420677:83 , srv1620622266:83 , srv167395310:83 , srv1811421904:83 , srv1948055539:83 , srv2143154442:83 , srv444742682:83 , srv88886804:83 , srv991349757:83 , srv1081534190:84 , srv1115508933:84 , srv129885965:84 , srv1576444007:84 , srv184152351:84 , srv1880127238:84 , srv349780047:84 , srv369349487:84 , srv486768250:84 , srv791961024:84 , srv1050393381:85 , srv11280536:85 , srv1188261602:85 , srv1275206049:85 , srv1433137073:85 , srv1543691009:85 , srv1639036808:85 , srv1648235227:85 , srv1855687723:85 , srv1979796798:85 , srv2036783496:85 , srv207447048:85 , srv2110105774:85 , srv270864186:85 , srv453476374:85 , srv677179524:85 , srv714993098:85 , srv79873209:85 , srv809595277:85 , srv881856447:85 , srv1160458373:86 , srv1253302287:86 , srv1393238407:86 , srv1399720109:86 , srv1561600695:86 , srv1684478712:86 , srv1778344114:86 , srv1814284943:86 , srv1937862200:86 , srv1954651771:86 , srv317426992:86 , srv394960129:86 , srv660780393:86 , srv715025365:86 , srv893291821:86 , srv1057915999:87 , srv1195808758:87 , srv1240648072:87 , srv1418354654:87 , srv1531884867:87 , srv1741457984:87 , srv1853788461:87 , srv1911569027:87 , srv2108169847:87 , srv451371961:87 , srv486278932:87 , srv594072499:87 , srv878394429:87 , srv992370368:87 , srv1066963939:88 , srv1135013446:88 , srv1142056850:88 , srv120670372:88 , srv1246529318:88 , srv1592742384:88 , srv1886333806:88 , srv2046423020:88 , srv2119585016:88 , srv2124769815:88 , srv273548185:88 , srv350024984:88 , srv365752328:88 , srv951576667:88 , srv1228742511:89 , srv258608178:89 , srv555058365:89 , srv77085764:89 , srv1164448715:90 , srv1249168840:90 , srv1631883185:90 , srv1171046368:91 , srv1209007601:91 , srv1970686859:92 , srv1897759541:93 , srv551129837:93 , srv93092426:95 , srv300485011:96 , srv633727769:449 } 2023-07-18 03:53:14,448 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-18 03:53:14,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv93092426=96, srv677179524=85, srv1188261602=12, srv1811421904=42, srv2119585016=61, srv350024984=70, srv129885965=22, srv878394429=92, srv317426992=68, srv2108169847=59, srv951576667=97, srv992370368=99, srv2046423020=57, srv633727769=83, srv1050393381=0, srv11280536=6, srv453476374=76, srv486768250=78, srv1979796798=55, srv486278932=77, srv893291821=95, srv1171046368=11, srv1209007601=15, srv2036783496=56, srv120670372=14, srv369349487=72, srv1880127238=47, srv451371961=75, srv1531884867=28, srv1601420677=33, srv1057915999=1, srv1561600695=30, srv1249168840=19, srv207447048=58, srv1954651771=53, srv1886333806=48, srv273548185=66, srv1246529318=18, srv1066963939=2, srv77085764=88, srv1275206049=21, srv1160458373=9, srv1911569027=50, srv1778344114=41, srv270864186=65, srv88886804=94, srv258608178=64, srv167395310=38, srv1648235227=37, srv1081534190=3, srv1684478712=39, srv1853788461=45, srv2110105774=60, srv660780393=84, srv1592742384=32, srv1814284943=43, srv444742682=74, srv1897759541=49, srv2124769815=62, srv715025365=87, srv1142056850=8, srv1418354654=26, srv1631883185=35, srv1135013446=7, srv1937862200=51, srv1253302287=20, srv572057717=81, srv1970686859=54, srv1119820446=5, srv1399720109=24, srv991349757=98, srv1240648072=17, srv1576444007=31, srv1639036808=36, srv881856447=93, srv1433137073=27, srv394960129=73, srv1115508933=4, srv1741457984=40, srv551129837=79, srv791961024=89, srv79873209=90, srv714993098=86, srv1543691009=29, srv809595277=91, srv1620622266=34, srv1400622292=25, srv2143154442=63, srv1948055539=52, srv555058365=80, srv349780047=69, srv1855687723=46, srv365752328=71, srv1195808758=13, srv1228742511=16, srv184152351=44, srv300485011=67, srv1164448715=10, srv1393238407=23, srv594072499=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-18 03:53:14,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 03:53:14,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-18 03:53:14,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-18 03:53:14,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-18 03:53:14,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-18 03:53:14,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-18 03:53:14,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-18 03:53:14,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-18 03:53:14,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-18 03:53:14,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-18 03:53:14,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-18 03:53:14,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-18 03:53:14,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-18 03:53:14,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-18 03:53:14,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-18 03:53:14,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-18 03:53:14,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-18 03:53:14,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-18 03:53:14,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-18 03:53:14,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-18 03:53:14,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-18 03:53:14,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-18 03:53:14,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-18 03:53:14,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-18 03:53:14,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-18 03:53:14,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-18 03:53:14,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-18 03:53:14,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-18 03:53:14,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-18 03:53:14,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-18 03:53:14,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-18 03:53:14,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-18 03:53:14,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-18 03:53:14,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-18 03:53:14,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-18 03:53:14,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-18 03:53:14,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-18 03:53:14,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-18 03:53:14,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-18 03:53:14,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-18 03:53:14,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-18 03:53:14,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-18 03:53:14,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-18 03:53:14,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-18 03:53:14,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-18 03:53:14,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-18 03:53:14,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-18 03:53:14,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-18 03:53:14,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-18 03:53:14,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-18 03:53:14,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-18 03:53:14,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-18 03:53:14,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-18 03:53:14,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-18 03:53:14,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-18 03:53:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-18 03:53:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-18 03:53:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-18 03:53:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-18 03:53:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-18 03:53:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-18 03:53:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-18 03:53:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-18 03:53:14,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-18 03:53:14,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-18 03:53:14,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-18 03:53:14,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-18 03:53:14,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-18 03:53:14,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-18 03:53:14,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-18 03:53:14,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-18 03:53:14,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-18 03:53:14,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-18 03:53:14,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-18 03:53:14,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-18 03:53:14,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-18 03:53:14,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-18 03:53:14,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-18 03:53:14,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-18 03:53:14,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-18 03:53:14,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-18 03:53:14,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-18 03:53:14,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-18 03:53:14,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-18 03:53:14,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-18 03:53:14,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-18 03:53:14,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-18 03:53:14,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-18 03:53:14,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-18 03:53:14,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-18 03:53:14,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-18 03:53:14,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-18 03:53:14,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-18 03:53:14,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-18 03:53:14,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-18 03:53:14,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-18 03:53:14,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-18 03:53:14,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-18 03:53:14,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-18 03:53:14,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-18 03:53:14,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-18 03:53:14,707 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040398316092815724, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05401292608314097, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040398316092815724, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16966666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23841666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 03:53:14,707 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-18 03:53:14,708 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17464869320817594, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040398316092815724, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05401292608314097, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040398316092815724, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16966666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23841666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-18 03:53:15,087 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 580 ms to try 5000 different iterations. Found a solution that moves 2615 regions; Going from a computed imbalance of 0.17464869320817594 to a new imbalance of 0.014758941785940028. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021617472714782957); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034892813264459165, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29055555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021617472714782957); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 03:53:15,099 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv300485011:76 , srv93092426:77 , srv1209007601:78 , srv1897759541:80 , srv551129837:80 , srv1249168840:81 , srv1970686859:81 , srv1164448715:82 , srv1171046368:82 , srv1631883185:82 , srv1886333806:82 , srv258608178:82 , srv1066963939:83 , srv120670372:83 , srv2119585016:83 , srv350024984:83 , srv365752328:83 , srv594072499:83 , srv878394429:83 , srv1057915999:84 , srv1195808758:84 , srv1399720109:84 , srv1741457984:84 , srv1853788461:84 , srv1937862200:84 , srv2108169847:84 , srv317426992:84 , srv486278932:84 , srv714993098:84 , srv1135013446:85 , srv1142056850:85 , srv1228742511:85 , srv1240648072:85 , srv1253302287:85 , srv1531884867:85 , srv1543691009:85 , srv1592742384:85 , srv1648235227:85 , srv1684478712:85 , srv1814284943:85 , srv1911569027:85 , srv2046423020:85 , srv273548185:85 , srv77085764:85 , srv951576667:85 , srv1050393381:86 , srv1188261602:86 , srv1246529318:86 , srv129885965:86 , srv1418354654:86 , srv1561600695:86 , srv1601420677:86 , srv207447048:86 , srv2124769815:86 , srv660780393:86 , srv677179524:86 , srv715025365:86 , srv79873209:86 , srv991349757:86 , srv1081534190:87 , srv1275206049:87 , srv1639036808:87 , srv1880127238:87 , srv1954651771:87 , srv2036783496:87 , srv2110105774:87 , srv369349487:87 , srv451371961:87 , srv486768250:87 , srv555058365:87 , srv88886804:87 , srv893291821:87 , srv992370368:87 , srv11280536:88 , srv1160458373:88 , srv1778344114:88 , srv1811421904:88 , srv1948055539:88 , srv270864186:88 , srv394960129:88 , srv809595277:88 , srv1115508933:89 , srv1393238407:89 , srv1400622292:89 , srv1433137073:89 , srv1576444007:89 , srv1855687723:89 , srv1979796798:89 , srv2143154442:89 , srv349780047:89 , srv444742682:89 , srv453476374:89 , srv1119820446:90 , srv572057717:90 , srv791961024:90 , srv1620622266:91 , srv167395310:91 , srv184152351:91 , srv881856447:91 , srv633727769:517 } 2023-07-18 03:53:15,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-18 03:53:15,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv93092426=96, srv677179524=85, srv1188261602=12, srv1811421904=42, srv2119585016=61, srv350024984=70, srv129885965=22, srv878394429=92, srv317426992=68, srv2108169847=59, srv951576667=97, srv992370368=99, srv2046423020=57, srv633727769=83, srv1050393381=0, srv11280536=6, srv453476374=76, srv486768250=78, srv1979796798=55, srv486278932=77, srv893291821=95, srv1171046368=11, srv1209007601=15, srv2036783496=56, srv120670372=14, srv369349487=72, srv1880127238=47, srv451371961=75, srv1531884867=28, srv1601420677=33, srv1057915999=1, srv1561600695=30, srv1249168840=19, srv207447048=58, srv1954651771=53, srv1886333806=48, srv273548185=66, srv1246529318=18, srv1066963939=2, srv77085764=88, srv1275206049=21, srv1160458373=9, srv1911569027=50, srv1778344114=41, srv270864186=65, srv88886804=94, srv258608178=64, srv167395310=38, srv1648235227=37, srv1081534190=3, srv1684478712=39, srv1853788461=45, srv2110105774=60, srv660780393=84, srv1592742384=32, srv1814284943=43, srv444742682=74, srv1897759541=49, srv2124769815=62, srv715025365=87, srv1142056850=8, srv1418354654=26, srv1631883185=35, srv1135013446=7, srv1937862200=51, srv1253302287=20, srv572057717=81, srv1970686859=54, srv1119820446=5, srv1399720109=24, srv991349757=98, srv1240648072=17, srv1576444007=31, srv1639036808=36, srv881856447=93, srv1433137073=27, srv394960129=73, srv1115508933=4, srv1741457984=40, srv551129837=79, srv791961024=89, srv79873209=90, srv714993098=86, srv1543691009=29, srv809595277=91, srv1620622266=34, srv1400622292=25, srv2143154442=63, srv1948055539=52, srv555058365=80, srv349780047=69, srv1855687723=46, srv365752328=71, srv1195808758=13, srv1228742511=16, srv184152351=44, srv300485011=67, srv1164448715=10, srv1393238407=23, srv594072499=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-18 03:53:15,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 03:53:15,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-18 03:53:15,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-18 03:53:15,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-18 03:53:15,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-18 03:53:15,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-18 03:53:15,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-18 03:53:15,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-18 03:53:15,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-18 03:53:15,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-18 03:53:15,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-18 03:53:15,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-18 03:53:15,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-18 03:53:15,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-18 03:53:15,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-18 03:53:15,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-18 03:53:15,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-18 03:53:15,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-18 03:53:15,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-18 03:53:15,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-18 03:53:15,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-18 03:53:15,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-18 03:53:15,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-18 03:53:15,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-18 03:53:15,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-18 03:53:15,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-18 03:53:15,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-18 03:53:15,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-18 03:53:15,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-18 03:53:15,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-18 03:53:15,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-18 03:53:15,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-18 03:53:15,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-18 03:53:15,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-18 03:53:15,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-18 03:53:15,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-18 03:53:15,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-18 03:53:15,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-18 03:53:15,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-18 03:53:15,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-18 03:53:15,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-18 03:53:15,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-18 03:53:15,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-18 03:53:15,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-18 03:53:15,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-18 03:53:15,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-18 03:53:15,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-18 03:53:15,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-18 03:53:15,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-18 03:53:15,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-18 03:53:15,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-18 03:53:15,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-18 03:53:15,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-18 03:53:15,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-18 03:53:15,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-18 03:53:15,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-18 03:53:15,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-18 03:53:15,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-18 03:53:15,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-18 03:53:15,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-18 03:53:15,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-18 03:53:15,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-18 03:53:15,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-18 03:53:15,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-18 03:53:15,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-18 03:53:15,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-18 03:53:15,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-18 03:53:15,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-18 03:53:15,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-18 03:53:15,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-18 03:53:15,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-18 03:53:15,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-18 03:53:15,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-18 03:53:15,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-18 03:53:15,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-18 03:53:15,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-18 03:53:15,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-18 03:53:15,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-18 03:53:15,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-18 03:53:15,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-18 03:53:15,123 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-18 03:53:15,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-18 03:53:15,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-18 03:53:15,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-18 03:53:15,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-18 03:53:15,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-18 03:53:15,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-18 03:53:15,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-18 03:53:15,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-18 03:53:15,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-18 03:53:15,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-18 03:53:15,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-18 03:53:15,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-18 03:53:15,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-18 03:53:15,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-18 03:53:15,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-18 03:53:15,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-18 03:53:15,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-18 03:53:15,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-18 03:53:15,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-18 03:53:15,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-18 03:53:15,326 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021617472714782957); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034892813264459165, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021617472714782957); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 03:53:15,326 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-18 03:53:15,326 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014740631164579271, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021617472714782957); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034892813264459165, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021617472714782957); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-18 03:53:15,655 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 506 ms to try 5000 different iterations. Found a solution that moves 910 regions; Going from a computed imbalance of 0.014740631164579271 to a new imbalance of 0.001032063732334296. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008236436283593619); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.012381877347325806); MoveCostFunction : (multiplier=7.0, imbalance=0.10111111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008236436283593619); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 03:53:15,658 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1393238407:81 , srv1160458373:82 , srv1246529318:82 , srv555058365:82 , srv881856447:82 , srv1228742511:83 , srv1433137073:83 , srv1620622266:83 , srv167395310:83 , srv184152351:83 , srv1979796798:83 , srv2124769815:83 , srv451371961:83 , srv453476374:83 , srv77085764:83 , srv992370368:83 , srv1115508933:84 , srv11280536:84 , srv1135013446:84 , srv1142056850:84 , srv1418354654:84 , srv1576444007:84 , srv1592742384:84 , srv1778344114:84 , srv1855687723:84 , srv1954651771:84 , srv2046423020:84 , srv394960129:84 , srv551129837:84 , srv791961024:84 , srv809595277:84 , srv1057915999:85 , srv1066963939:85 , srv1119820446:85 , srv1171046368:85 , srv1195808758:85 , srv1240648072:85 , srv1275206049:85 , srv1400622292:85 , srv1561600695:85 , srv1639036808:85 , srv1853788461:85 , srv1897759541:85 , srv1970686859:85 , srv2036783496:85 , srv2110105774:85 , srv2143154442:85 , srv270864186:85 , srv273548185:85 , srv349780047:85 , srv369349487:85 , srv486768250:85 , srv572057717:85 , srv660780393:85 , srv677179524:85 , srv715025365:85 , srv893291821:85 , srv93092426:85 , srv951576667:85 , srv1164448715:86 , srv1249168840:86 , srv129885965:86 , srv1531884867:86 , srv1741457984:86 , srv1814284943:86 , srv1911569027:86 , srv2108169847:86 , srv2119585016:86 , srv300485011:86 , srv350024984:86 , srv444742682:86 , srv486278932:86 , srv594072499:86 , srv79873209:86 , srv1050393381:87 , srv1081534190:87 , srv1188261602:87 , srv120670372:87 , srv1209007601:87 , srv1253302287:87 , srv1543691009:87 , srv1631883185:87 , srv1684478712:87 , srv1811421904:87 , srv1880127238:87 , srv1886333806:87 , srv1937862200:87 , srv1948055539:87 , srv207447048:87 , srv258608178:87 , srv878394429:87 , srv991349757:87 , srv1399720109:88 , srv1601420677:88 , srv1648235227:88 , srv317426992:88 , srv714993098:88 , srv88886804:88 , srv365752328:89 , srv633727769:565 } 2023-07-18 03:53:15,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-18 03:53:15,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv93092426=96, srv677179524=85, srv1188261602=12, srv1811421904=42, srv2119585016=61, srv350024984=70, srv129885965=22, srv878394429=92, srv317426992=68, srv2108169847=59, srv951576667=97, srv992370368=99, srv2046423020=57, srv633727769=83, srv1050393381=0, srv11280536=6, srv453476374=76, srv486768250=78, srv1979796798=55, srv486278932=77, srv893291821=95, srv1171046368=11, srv1209007601=15, srv2036783496=56, srv120670372=14, srv369349487=72, srv1880127238=47, srv451371961=75, srv1531884867=28, srv1601420677=33, srv1057915999=1, srv1561600695=30, srv1249168840=19, srv207447048=58, srv1954651771=53, srv1886333806=48, srv273548185=66, srv1246529318=18, srv1066963939=2, srv77085764=88, srv1275206049=21, srv1160458373=9, srv1911569027=50, srv1778344114=41, srv270864186=65, srv88886804=94, srv258608178=64, srv167395310=38, srv1648235227=37, srv1081534190=3, srv1684478712=39, srv1853788461=45, srv2110105774=60, srv660780393=84, srv1592742384=32, srv1814284943=43, srv444742682=74, srv1897759541=49, srv2124769815=62, srv715025365=87, srv1142056850=8, srv1418354654=26, srv1631883185=35, srv1135013446=7, srv1937862200=51, srv1253302287=20, srv572057717=81, srv1970686859=54, srv1119820446=5, srv1399720109=24, srv991349757=98, srv1240648072=17, srv1576444007=31, srv1639036808=36, srv881856447=93, srv1433137073=27, srv394960129=73, srv1115508933=4, srv1741457984=40, srv551129837=79, srv791961024=89, srv79873209=90, srv714993098=86, srv1543691009=29, srv809595277=91, srv1620622266=34, srv1400622292=25, srv2143154442=63, srv1948055539=52, srv555058365=80, srv349780047=69, srv1855687723=46, srv365752328=71, srv1195808758=13, srv1228742511=16, srv184152351=44, srv300485011=67, srv1164448715=10, srv1393238407=23, srv594072499=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-18 03:53:15,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 03:53:15,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-18 03:53:15,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-18 03:53:15,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-18 03:53:15,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-18 03:53:15,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-18 03:53:15,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-18 03:53:15,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-18 03:53:15,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-18 03:53:15,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-18 03:53:15,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-18 03:53:15,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-18 03:53:15,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-18 03:53:15,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-18 03:53:15,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-18 03:53:15,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-18 03:53:15,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-18 03:53:15,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-18 03:53:15,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-18 03:53:15,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-18 03:53:15,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-18 03:53:15,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-18 03:53:15,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-18 03:53:15,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-18 03:53:15,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-18 03:53:15,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-18 03:53:15,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-18 03:53:15,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-18 03:53:15,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-18 03:53:15,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-18 03:53:15,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-18 03:53:15,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-18 03:53:15,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-18 03:53:15,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-18 03:53:15,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-18 03:53:15,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-18 03:53:15,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-18 03:53:15,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-18 03:53:15,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-18 03:53:15,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-18 03:53:15,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-18 03:53:15,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-18 03:53:15,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-18 03:53:15,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-18 03:53:15,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-18 03:53:15,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-18 03:53:15,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-18 03:53:15,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-18 03:53:15,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-18 03:53:15,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-18 03:53:15,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-18 03:53:15,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-18 03:53:15,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-18 03:53:15,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-18 03:53:15,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-18 03:53:15,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-18 03:53:15,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-18 03:53:15,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-18 03:53:15,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-18 03:53:15,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-18 03:53:15,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-18 03:53:15,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-18 03:53:15,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-18 03:53:15,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-18 03:53:15,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-18 03:53:15,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-18 03:53:15,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-18 03:53:15,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-18 03:53:15,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-18 03:53:15,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-18 03:53:15,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-18 03:53:15,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-18 03:53:15,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-18 03:53:15,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-18 03:53:15,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-18 03:53:15,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-18 03:53:15,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-18 03:53:15,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-18 03:53:15,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-18 03:53:15,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-18 03:53:15,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-18 03:53:15,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-18 03:53:15,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-18 03:53:15,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-18 03:53:15,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-18 03:53:15,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-18 03:53:15,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-18 03:53:15,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-18 03:53:15,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-18 03:53:15,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-18 03:53:15,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-18 03:53:15,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-18 03:53:15,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-18 03:53:15,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-18 03:53:15,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-18 03:53:15,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-18 03:53:15,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-18 03:53:15,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-18 03:53:15,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-18 03:53:15,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-18 03:53:15,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-18 03:53:15,966 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008236436283593619); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.012381877347325806); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008236436283593619); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 03:53:15,966 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-18 03:53:15,968 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0010256917761437454, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008236436283593619); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.012381877347325806); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008236436283593619); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-18 03:53:16,360 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 648 ms to try 5000 different iterations. Found a solution that moves 306 regions; Going from a computed imbalance of 0.0010256917761437454 to a new imbalance of 4.3087902816507984E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.034, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 03:53:16,362 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv365752328:81 , srv1631883185:82 , srv1081534190:83 , srv1880127238:83 , srv893291821:83 , srv1050393381:84 , srv1119820446:84 , srv1164448715:84 , srv1171046368:84 , srv1188261602:84 , srv120670372:84 , srv1253302287:84 , srv1399720109:84 , srv1648235227:84 , srv1684478712:84 , srv1814284943:84 , srv1855687723:84 , srv1897759541:84 , srv1911569027:84 , srv1937862200:84 , srv1948055539:84 , srv207447048:84 , srv2143154442:84 , srv258608178:84 , srv273548185:84 , srv300485011:84 , srv349780047:84 , srv394960129:84 , srv444742682:84 , srv555058365:84 , srv791961024:84 , srv881856447:84 , srv88886804:84 , srv951576667:84 , srv1115508933:85 , srv11280536:85 , srv1135013446:85 , srv1142056850:85 , srv1195808758:85 , srv1228742511:85 , srv1240648072:85 , srv1249168840:85 , srv1275206049:85 , srv1418354654:85 , srv1433137073:85 , srv1531884867:85 , srv1543691009:85 , srv1561600695:85 , srv1576444007:85 , srv1592742384:85 , srv1601420677:85 , srv1620622266:85 , srv1639036808:85 , srv167395310:85 , srv1778344114:85 , srv1811421904:85 , srv184152351:85 , srv1886333806:85 , srv1954651771:85 , srv1970686859:85 , srv1979796798:85 , srv2036783496:85 , srv2046423020:85 , srv2108169847:85 , srv2110105774:85 , srv2119585016:85 , srv2124769815:85 , srv270864186:85 , srv317426992:85 , srv350024984:85 , srv451371961:85 , srv453476374:85 , srv486278932:85 , srv551129837:85 , srv572057717:85 , srv714993098:85 , srv715025365:85 , srv77085764:85 , srv79873209:85 , srv809595277:85 , srv878394429:85 , srv93092426:85 , srv992370368:85 , srv1057915999:86 , srv1066963939:86 , srv1160458373:86 , srv1209007601:86 , srv1246529318:86 , srv129885965:86 , srv1393238407:86 , srv1400622292:86 , srv1741457984:86 , srv1853788461:86 , srv369349487:86 , srv486768250:86 , srv594072499:86 , srv660780393:86 , srv677179524:86 , srv991349757:86 , srv633727769:611 } 2023-07-18 03:53:16,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-18 03:53:16,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv93092426=96, srv677179524=85, srv1188261602=12, srv1811421904=42, srv2119585016=61, srv350024984=70, srv129885965=22, srv878394429=92, srv317426992=68, srv2108169847=59, srv951576667=97, srv992370368=99, srv2046423020=57, srv633727769=83, srv1050393381=0, srv11280536=6, srv453476374=76, srv486768250=78, srv1979796798=55, srv486278932=77, srv893291821=95, srv1171046368=11, srv1209007601=15, srv2036783496=56, srv120670372=14, srv369349487=72, srv1880127238=47, srv451371961=75, srv1531884867=28, srv1601420677=33, srv1057915999=1, srv1561600695=30, srv1249168840=19, srv207447048=58, srv1954651771=53, srv1886333806=48, srv273548185=66, srv1246529318=18, srv1066963939=2, srv77085764=88, srv1275206049=21, srv1160458373=9, srv1911569027=50, srv1778344114=41, srv270864186=65, srv88886804=94, srv258608178=64, srv167395310=38, srv1648235227=37, srv1081534190=3, srv1684478712=39, srv1853788461=45, srv2110105774=60, srv660780393=84, srv1592742384=32, srv1814284943=43, srv444742682=74, srv1897759541=49, srv2124769815=62, srv715025365=87, srv1142056850=8, srv1418354654=26, srv1631883185=35, srv1135013446=7, srv1937862200=51, srv1253302287=20, srv572057717=81, srv1970686859=54, srv1119820446=5, srv1399720109=24, srv991349757=98, srv1240648072=17, srv1576444007=31, srv1639036808=36, srv881856447=93, srv1433137073=27, srv394960129=73, srv1115508933=4, srv1741457984=40, srv551129837=79, srv791961024=89, srv79873209=90, srv714993098=86, srv1543691009=29, srv809595277=91, srv1620622266=34, srv1400622292=25, srv2143154442=63, srv1948055539=52, srv555058365=80, srv349780047=69, srv1855687723=46, srv365752328=71, srv1195808758=13, srv1228742511=16, srv184152351=44, srv300485011=67, srv1164448715=10, srv1393238407=23, srv594072499=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-18 03:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 03:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-18 03:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-18 03:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-18 03:53:16,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-18 03:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-18 03:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-18 03:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-18 03:53:16,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-18 03:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-18 03:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-18 03:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-18 03:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-18 03:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-18 03:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-18 03:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-18 03:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-18 03:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-18 03:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-18 03:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-18 03:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-18 03:53:16,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-18 03:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-18 03:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-18 03:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-18 03:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-18 03:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-18 03:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-18 03:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-18 03:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-18 03:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-18 03:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-18 03:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-18 03:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-18 03:53:16,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-18 03:53:16,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-18 03:53:16,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-18 03:53:16,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-18 03:53:16,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-18 03:53:16,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-18 03:53:16,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-18 03:53:16,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-18 03:53:16,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-18 03:53:16,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-18 03:53:16,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-18 03:53:16,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-18 03:53:16,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-18 03:53:16,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-18 03:53:16,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-18 03:53:16,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-18 03:53:16,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-18 03:53:16,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-18 03:53:16,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-18 03:53:16,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-18 03:53:16,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-18 03:53:16,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-18 03:53:16,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-18 03:53:16,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-18 03:53:16,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-18 03:53:16,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-18 03:53:16,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-18 03:53:16,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-18 03:53:16,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-18 03:53:16,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-18 03:53:16,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-18 03:53:16,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-18 03:53:16,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-18 03:53:16,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-18 03:53:16,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-18 03:53:16,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-18 03:53:16,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-18 03:53:16,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-18 03:53:16,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-18 03:53:16,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-18 03:53:16,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-18 03:53:16,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-18 03:53:16,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-18 03:53:16,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-18 03:53:16,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-18 03:53:16,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-18 03:53:16,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-18 03:53:16,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-18 03:53:16,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-18 03:53:16,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-18 03:53:16,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-18 03:53:16,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-18 03:53:16,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-18 03:53:16,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-18 03:53:16,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-18 03:53:16,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-18 03:53:16,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-18 03:53:16,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-18 03:53:16,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-18 03:53:16,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-18 03:53:16,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-18 03:53:16,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-18 03:53:16,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-18 03:53:16,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-18 03:53:16,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-18 03:53:16,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-18 03:53:16,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-18 03:53:16,562 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 03:53:16,562 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-07-18 03:53:16,562 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.0945245020564624E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-07-18 03:53:16,908 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 487 ms to try 5000 different iterations. Found a solution that moves 28 regions; Going from a computed imbalance of 4.0945245020564624E-5 to a new imbalance of 7.269299486807925E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.003111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 03:53:16,909 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1400622292:83 , srv1531884867:83 , srv1741457984:83 , srv1778344114:83 , srv1811421904:83 , srv184152351:83 , srv1970686859:83 , srv270864186:83 , srv317426992:83 , srv660780393:83 , srv1050393381:84 , srv1057915999:84 , srv1066963939:84 , srv1115508933:84 , srv1119820446:84 , srv11280536:84 , srv1135013446:84 , srv1142056850:84 , srv1160458373:84 , srv1164448715:84 , srv1171046368:84 , srv1188261602:84 , srv120670372:84 , srv1209007601:84 , srv1228742511:84 , srv1240648072:84 , srv1246529318:84 , srv1249168840:84 , srv1253302287:84 , srv1275206049:84 , srv129885965:84 , srv1393238407:84 , srv1399720109:84 , srv1418354654:84 , srv1433137073:84 , srv1543691009:84 , srv1561600695:84 , srv1576444007:84 , srv1592742384:84 , srv1601420677:84 , srv1620622266:84 , srv1639036808:84 , srv1648235227:84 , srv167395310:84 , srv1684478712:84 , srv1853788461:84 , srv1855687723:84 , srv1886333806:84 , srv1897759541:84 , srv1911569027:84 , srv1948055539:84 , srv1954651771:84 , srv1979796798:84 , srv2036783496:84 , srv2046423020:84 , srv207447048:84 , srv2108169847:84 , srv2110105774:84 , srv2119585016:84 , srv2124769815:84 , srv258608178:84 , srv273548185:84 , srv300485011:84 , srv349780047:84 , srv350024984:84 , srv369349487:84 , srv394960129:84 , srv444742682:84 , srv451371961:84 , srv453476374:84 , srv486278932:84 , srv486768250:84 , srv551129837:84 , srv555058365:84 , srv594072499:84 , srv677179524:84 , srv714993098:84 , srv715025365:84 , srv77085764:84 , srv791961024:84 , srv79873209:84 , srv809595277:84 , srv878394429:84 , srv881856447:84 , srv88886804:84 , srv93092426:84 , srv951576667:84 , srv991349757:84 , srv992370368:84 , srv1081534190:85 , srv1195808758:85 , srv1631883185:85 , srv1814284943:85 , srv1880127238:85 , srv1937862200:85 , srv2143154442:85 , srv365752328:85 , srv572057717:85 , srv893291821:85 , srv633727769:684 } 2023-07-18 03:53:16,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-18 03:53:16,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv93092426=96, srv677179524=85, srv1188261602=12, srv1811421904=42, srv2119585016=61, srv350024984=70, srv129885965=22, srv878394429=92, srv317426992=68, srv2108169847=59, srv951576667=97, srv992370368=99, srv2046423020=57, srv633727769=83, srv1050393381=0, srv11280536=6, srv453476374=76, srv486768250=78, srv1979796798=55, srv486278932=77, srv893291821=95, srv1171046368=11, srv1209007601=15, srv2036783496=56, srv120670372=14, srv369349487=72, srv1880127238=47, srv451371961=75, srv1531884867=28, srv1601420677=33, srv1057915999=1, srv1561600695=30, srv1249168840=19, srv207447048=58, srv1954651771=53, srv1886333806=48, srv273548185=66, srv1246529318=18, srv1066963939=2, srv77085764=88, srv1275206049=21, srv1160458373=9, srv1911569027=50, srv1778344114=41, srv270864186=65, srv88886804=94, srv258608178=64, srv167395310=38, srv1648235227=37, srv1081534190=3, srv1684478712=39, srv1853788461=45, srv2110105774=60, srv660780393=84, srv1592742384=32, srv1814284943=43, srv444742682=74, srv1897759541=49, srv2124769815=62, srv715025365=87, srv1142056850=8, srv1418354654=26, srv1631883185=35, srv1135013446=7, srv1937862200=51, srv1253302287=20, srv572057717=81, srv1970686859=54, srv1119820446=5, srv1399720109=24, srv991349757=98, srv1240648072=17, srv1576444007=31, srv1639036808=36, srv881856447=93, srv1433137073=27, srv394960129=73, srv1115508933=4, srv1741457984=40, srv551129837=79, srv791961024=89, srv79873209=90, srv714993098=86, srv1543691009=29, srv809595277=91, srv1620622266=34, srv1400622292=25, srv2143154442=63, srv1948055539=52, srv555058365=80, srv349780047=69, srv1855687723=46, srv365752328=71, srv1195808758=13, srv1228742511=16, srv184152351=44, srv300485011=67, srv1164448715=10, srv1393238407=23, srv594072499=82} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-18 03:53:16,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 03:53:16,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-18 03:53:16,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-18 03:53:16,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-18 03:53:16,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-18 03:53:16,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-18 03:53:16,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-18 03:53:16,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-18 03:53:16,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-18 03:53:16,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-18 03:53:16,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-18 03:53:16,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-18 03:53:16,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-18 03:53:16,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-18 03:53:16,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-18 03:53:16,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-18 03:53:16,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-18 03:53:16,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-18 03:53:16,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-18 03:53:16,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-18 03:53:16,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-18 03:53:16,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-18 03:53:16,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-18 03:53:16,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-18 03:53:16,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-18 03:53:16,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-18 03:53:16,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-18 03:53:16,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-18 03:53:16,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-18 03:53:16,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-18 03:53:16,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-07-18 03:53:16,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-07-18 03:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-07-18 03:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-07-18 03:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-07-18 03:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-07-18 03:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-07-18 03:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-07-18 03:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-07-18 03:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-07-18 03:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-07-18 03:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-07-18 03:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-07-18 03:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-07-18 03:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-07-18 03:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-07-18 03:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-07-18 03:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-07-18 03:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-07-18 03:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-07-18 03:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-07-18 03:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-07-18 03:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-07-18 03:53:16,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-07-18 03:53:16,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-07-18 03:53:16,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-07-18 03:53:16,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-07-18 03:53:16,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-07-18 03:53:16,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-07-18 03:53:16,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-07-18 03:53:16,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-07-18 03:53:16,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-07-18 03:53:16,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-07-18 03:53:16,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-07-18 03:53:16,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-07-18 03:53:16,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-07-18 03:53:16,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-07-18 03:53:16,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-07-18 03:53:16,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-07-18 03:53:16,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-07-18 03:53:16,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-07-18 03:53:16,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-07-18 03:53:16,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-07-18 03:53:16,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-07-18 03:53:16,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-07-18 03:53:16,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-07-18 03:53:16,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-07-18 03:53:16,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-07-18 03:53:16,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-07-18 03:53:17,058 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 03:53:17,058 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1400622292:83 , srv1531884867:83 , srv1741457984:83 , srv1778344114:83 , srv1811421904:83 , srv184152351:83 , srv1970686859:83 , srv270864186:83 , srv317426992:83 , srv660780393:83 , srv1050393381:84 , srv1057915999:84 , srv1066963939:84 , srv1115508933:84 , srv1119820446:84 , srv11280536:84 , srv1135013446:84 , srv1142056850:84 , srv1160458373:84 , srv1164448715:84 , srv1171046368:84 , srv1188261602:84 , srv120670372:84 , srv1209007601:84 , srv1228742511:84 , srv1240648072:84 , srv1246529318:84 , srv1249168840:84 , srv1253302287:84 , srv1275206049:84 , srv129885965:84 , srv1393238407:84 , srv1399720109:84 , srv1418354654:84 , srv1433137073:84 , srv1543691009:84 , srv1561600695:84 , srv1576444007:84 , srv1592742384:84 , srv1601420677:84 , srv1620622266:84 , srv1639036808:84 , srv1648235227:84 , srv167395310:84 , srv1684478712:84 , srv1853788461:84 , srv1855687723:84 , srv1886333806:84 , srv1897759541:84 , srv1911569027:84 , srv1948055539:84 , srv1954651771:84 , srv1979796798:84 , srv2036783496:84 , srv2046423020:84 , srv207447048:84 , srv2108169847:84 , srv2110105774:84 , srv2119585016:84 , srv2124769815:84 , srv258608178:84 , srv273548185:84 , srv300485011:84 , srv349780047:84 , srv350024984:84 , srv369349487:84 , srv394960129:84 , srv444742682:84 , srv451371961:84 , srv453476374:84 , srv486278932:84 , srv486768250:84 , srv551129837:84 , srv555058365:84 , srv594072499:84 , srv677179524:84 , srv714993098:84 , srv715025365:84 , srv77085764:84 , srv791961024:84 , srv79873209:84 , srv809595277:84 , srv878394429:84 , srv881856447:84 , srv88886804:84 , srv93092426:84 , srv951576667:84 , srv991349757:84 , srv992370368:84 , srv1081534190:85 , srv1195808758:85 , srv1631883185:85 , srv1814284943:85 , srv1880127238:85 , srv1937862200:85 , srv2143154442:85 , srv365752328:85 , srv572057717:85 , srv893291821:85 , srv633727769:684 } 2023-07-18 03:53:17,087 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=328 (was 328), ProcessCount=173 (was 173), AvailableMemoryMB=6665 (was 7540) 2023-07-18 03:53:17,098 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=328, ProcessCount=173, AvailableMemoryMB=6665 2023-07-18 03:53:17,099 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-07-18 03:53:17,099 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-07-18 03:53:17,109 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv107721293:84 , srv107882301:84 , srv1102041530:84 , srv1233144017:84 , srv1248570007:84 , srv1276074824:84 , srv128377946:84 , srv1308804640:84 , srv1330633473:84 , srv1387211707:84 , srv1461855059:84 , srv1495387981:84 , srv1622958348:84 , srv1704941712:84 , srv1757432404:84 , srv1872235366:84 , srv1886157838:84 , srv1925865897:84 , srv204269848:84 , srv2098365829:84 , srv290276410:84 , srv380234191:84 , srv426284139:84 , srv564352954:84 , srv598407556:84 , srv729817678:84 , srv764107611:84 , srv799070123:84 , srv953231295:84 , srv1098578626:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-07-18 03:53:17,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-18 03:53:17,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1276074824=6, srv2098365829=20, srv1102041530=3, srv290276410=21, srv107721293=0, srv1098578626=2, srv953231295=29, srv1925865897=18, srv564352954=24, srv1330633473=9, srv799070123=28, srv1704941712=14, srv1886157838=17, srv598407556=25, srv1757432404=15, srv107882301=1, srv1872235366=16, srv1308804640=8, srv1387211707=10, srv1233144017=4, srv1461855059=11, srv1495387981=12, srv1622958348=13, srv729817678=26, srv380234191=22, srv764107611=27, srv128377946=7, srv1248570007=5, srv204269848=19, srv426284139=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-18 03:53:17,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 03:53:17,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-18 03:53:17,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-18 03:53:17,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-18 03:53:17,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-18 03:53:17,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-18 03:53:17,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-18 03:53:17,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-18 03:53:17,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-18 03:53:17,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-18 03:53:17,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-18 03:53:17,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-18 03:53:17,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-18 03:53:17,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-18 03:53:17,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-18 03:53:17,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-18 03:53:17,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-18 03:53:17,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-18 03:53:17,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-18 03:53:17,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-18 03:53:17,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-18 03:53:17,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-18 03:53:17,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-18 03:53:17,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-18 03:53:17,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-18 03:53:17,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-18 03:53:17,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-18 03:53:17,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-18 03:53:17,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-18 03:53:17,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-18 03:53:17,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-07-18 03:53:17,149 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1285554949173606, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 03:53:17,149 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912598117431042, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1285554949173606, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-07-18 03:55:17,130 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5495032 different iterations. Found a solution that moves 1852 regions; Going from a computed imbalance of 0.9912598117431042 to a new imbalance of 4.322660390073086E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6859259259259259, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 03:55:17,136 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv107721293:90 , srv107882301:90 , srv1098578626:90 , srv1102041530:90 , srv1233144017:90 , srv1248570007:90 , srv1276074824:90 , srv128377946:90 , srv1308804640:90 , srv1330633473:90 , srv1387211707:90 , srv1461855059:90 , srv1495387981:90 , srv1622958348:90 , srv1704941712:90 , srv1757432404:90 , srv1872235366:90 , srv1886157838:90 , srv1925865897:90 , srv204269848:90 , srv2098365829:90 , srv290276410:90 , srv380234191:90 , srv426284139:90 , srv564352954:90 , srv598407556:90 , srv729817678:90 , srv764107611:90 , srv799070123:90 , srv953231295:90 } 2023-07-18 03:55:17,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-07-18 03:55:17,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1276074824=6, srv2098365829=20, srv1102041530=3, srv290276410=21, srv107721293=0, srv1098578626=2, srv953231295=29, srv1925865897=18, srv564352954=24, srv1330633473=9, srv799070123=28, srv1704941712=14, srv1886157838=17, srv598407556=25, srv1757432404=15, srv107882301=1, srv1872235366=16, srv1308804640=8, srv1387211707=10, srv1233144017=4, srv1461855059=11, srv1495387981=12, srv1622958348=13, srv729817678=26, srv380234191=22, srv764107611=27, srv128377946=7, srv1248570007=5, srv204269848=19, srv426284139=23} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-07-18 03:55:17,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-18 03:55:17,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-07-18 03:55:17,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-07-18 03:55:17,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-07-18 03:55:17,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-07-18 03:55:17,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-07-18 03:55:17,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-07-18 03:55:17,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-07-18 03:55:17,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-07-18 03:55:17,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-07-18 03:55:17,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-07-18 03:55:17,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-07-18 03:55:17,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-07-18 03:55:17,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-07-18 03:55:17,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-07-18 03:55:17,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-07-18 03:55:17,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-07-18 03:55:17,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-07-18 03:55:17,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-07-18 03:55:17,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-07-18 03:55:17,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-07-18 03:55:17,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-07-18 03:55:17,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-07-18 03:55:17,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-07-18 03:55:17,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-07-18 03:55:17,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-07-18 03:55:17,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-07-18 03:55:17,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-07-18 03:55:17,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-07-18 03:55:17,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-07-18 03:55:17,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-07-18 03:55:17,154 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-07-18 03:55:17,154 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv107721293:90 , srv107882301:90 , srv1098578626:90 , srv1102041530:90 , srv1233144017:90 , srv1248570007:90 , srv1276074824:90 , srv128377946:90 , srv1308804640:90 , srv1330633473:90 , srv1387211707:90 , srv1461855059:90 , srv1495387981:90 , srv1622958348:90 , srv1704941712:90 , srv1757432404:90 , srv1872235366:90 , srv1886157838:90 , srv1925865897:90 , srv204269848:90 , srv2098365829:90 , srv290276410:90 , srv380234191:90 , srv426284139:90 , srv564352954:90 , srv598407556:90 , srv729817678:90 , srv764107611:90 , srv799070123:90 , srv953231295:90 } 2023-07-18 03:55:17,168 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=210 (was 328), ProcessCount=172 (was 173), AvailableMemoryMB=5813 (was 6665)