2023-04-03 04:56:28,001 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-04-03 04:56:28,378 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-04-03 04:56:28,495 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-04-03 04:56:28,555 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-03 04:56:28,563 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-03 04:56:28,576 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=442, ProcessCount=169, AvailableMemoryMB=4681
2023-04-03 04:56:28,578 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-03 04:56:28,578 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-03 04:56:28,700 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1068601747:84 , srv1083864939:84 , srv111345863:84 , srv1149941902:84 , srv1156475621:84 , srv1159350049:84 , srv117290899:84 , srv1185171544:84 , srv1222683308:84 , srv1259773114:84 , srv1283919546:84 , srv1291197943:84 , srv1293504679:84 , srv133452944:84 , srv1346813548:84 , srv1405306101:84 , srv1463006174:84 , srv1472097678:84 , srv1474667701:84 , srv1480823196:84 , srv1492525003:84 , srv1498606686:84 , srv1499456817:84 , srv1521614752:84 , srv154102820:84 , srv156215139:84 , srv1590551711:84 , srv1661510174:84 , srv1669099576:84 , srv1703177380:84 , srv171362798:84 , srv1732411036:84 , srv1751259047:84 , srv1757174813:84 , srv179276702:84 , srv1804950666:84 , srv18112717:84 , srv1819961710:84 , srv1830635479:84 , srv1843261339:84 , srv1874381902:84 , srv1886282596:84 , srv1891922563:84 , srv1908784337:84 , srv1914302063:84 , srv1916938612:84 , srv1917133390:84 , srv1930559648:84 , srv1949815666:84 , srv1965759903:84 , srv1984254309:84 , srv2006612521:84 , srv2034871029:84 , srv2066008166:84 , srv2080702927:84 , srv2146399080:84 , srv225877013:84 , srv232857239:84 , srv239355388:84 , srv276717301:84 , srv284633849:84 , srv29857683:84 , srv325321771:84 , srv326813491:84 , srv346128466:84 , srv37498204:84 , srv375582984:84 , srv384392594:84 , srv391475836:84 , srv39387182:84 , srv434677467:84 , srv438503280:84 , srv440803995:84 , srv450756119:84 , srv464234683:84 , srv505439752:84 , srv520997922:84 , srv575276354:84 , srv597606703:84 , srv62057421:84 , srv630211292:84 , srv725568652:84 , srv747977591:84 , srv751925619:84 , srv81937387:84 , srv830180987:84 , srv86169944:84 , srv864106146:84 , srv899079640:84 , srv909483782:84 , srv918290462:84 , srv947164576:84 , srv956931750:84 , srv973346738:84 , srv981746:84 , srv985330021:84 , srv988638772:84 , srv995655787:84 , srv997437701:84 , srv894909235:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-04-03 04:56:28,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-03 04:56:28,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2080702927=54, srv725568652=81, srv276717301=59, srv440803995=72, srv1498606686=21, srv1293504679=12, srv284633849=60, srv988638772=97, srv434677467=70, srv326813491=63, srv1492525003=20, srv117290899=6, srv1149941902=3, srv995655787=98, srv918290462=91, srv973346738=94, srv1283919546=10, srv747977591=82, srv1732411036=31, srv39387182=69, srv154102820=24, srv2146399080=55, srv1874381902=40, srv956931750=93, srv391475836=68, srv1156475621=4, srv156215139=25, srv29857683=61, srv997437701=99, srv171362798=30, srv81937387=84, srv1917133390=46, srv630211292=80, srv1965759903=49, srv225877013=56, srv179276702=34, srv830180987=85, srv985330021=96, srv1480823196=19, srv1521614752=23, srv909483782=90, srv864106146=87, srv1185171544=7, srv1804950666=35, srv1259773114=9, srv384392594=67, srv450756119=73, srv62057421=79, srv1463006174=16, srv1068601747=0, srv1886282596=41, srv575276354=77, srv111345863=2, srv232857239=57, srv1346813548=14, srv1830635479=38, srv1405306101=15, srv239355388=58, srv1843261339=39, srv1159350049=5, srv133452944=13, srv37498204=65, srv346128466=64, srv1499456817=22, srv1908784337=43, srv1291197943=11, srv1669099576=28, srv1472097678=17, srv1474667701=18, srv505439752=75, srv894909235=88, srv947164576=92, srv1949815666=48, srv1661510174=27, srv1751259047=32, srv1757174813=33, srv1083864939=1, srv2006612521=51, srv438503280=71, srv1590551711=26, srv86169944=86, srv899079640=89, srv520997922=76, srv1222683308=8, srv1914302063=44, srv1930559648=47, srv325321771=62, srv597606703=78, srv751925619=83, srv1703177380=29, srv2034871029=52, srv2066008166=53, srv375582984=66, srv981746=95, srv1984254309=50, srv18112717=36, srv1819961710=37, srv1891922563=42, srv1916938612=45, srv464234683=74} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-03 04:56:28,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-03 04:56:28,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-03 04:56:28,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-03 04:56:28,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-03 04:56:28,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-03 04:56:28,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-03 04:56:28,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-03 04:56:28,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-03 04:56:28,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-03 04:56:28,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-03 04:56:28,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-03 04:56:28,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-03 04:56:28,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-03 04:56:28,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-03 04:56:28,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-03 04:56:28,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-03 04:56:28,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-03 04:56:28,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-03 04:56:28,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-03 04:56:28,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-03 04:56:28,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-03 04:56:28,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-03 04:56:28,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-03 04:56:28,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-03 04:56:28,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-03 04:56:28,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-03 04:56:28,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-03 04:56:28,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-03 04:56:28,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-03 04:56:28,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-03 04:56:28,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-03 04:56:28,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-03 04:56:28,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-03 04:56:28,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-03 04:56:28,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-03 04:56:28,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-03 04:56:28,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-03 04:56:28,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-03 04:56:28,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-03 04:56:28,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-03 04:56:28,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-03 04:56:28,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-03 04:56:28,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-03 04:56:28,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-03 04:56:28,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-03 04:56:28,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-03 04:56:28,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-03 04:56:28,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-03 04:56:28,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-03 04:56:28,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-03 04:56:28,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-03 04:56:28,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-03 04:56:28,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-03 04:56:28,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-03 04:56:28,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-03 04:56:28,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-03 04:56:28,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-03 04:56:28,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-03 04:56:28,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-03 04:56:28,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-03 04:56:28,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-03 04:56:28,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-03 04:56:28,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-03 04:56:28,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-03 04:56:28,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-03 04:56:28,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-03 04:56:28,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-03 04:56:28,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-03 04:56:28,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-03 04:56:28,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-03 04:56:28,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-03 04:56:28,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-03 04:56:28,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-03 04:56:28,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-03 04:56:28,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-03 04:56:28,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-03 04:56:28,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-03 04:56:28,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-03 04:56:28,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-03 04:56:28,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-03 04:56:28,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-03 04:56:28,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-03 04:56:28,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-03 04:56:28,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-03 04:56:28,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-03 04:56:28,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-03 04:56:28,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-03 04:56:28,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-03 04:56:28,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-03 04:56:28,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-03 04:56:28,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-03 04:56:28,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-03 04:56:28,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-03 04:56:28,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-03 04:56:28,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-03 04:56:28,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-03 04:56:28,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-03 04:56:28,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-03 04:56:28,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-03 04:56:28,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-03 04:56:28,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-03 04:56:29,049 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 04:56:29,050 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-03 04:56:29,050 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-03 04:56:29,421 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 540 ms to try 5000 different iterations.  Found a solution that moves 4668 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.180883490999824. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03791221869299151, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054496985573406743, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5186666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03791221869299151, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1765, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23908333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 04:56:29,435 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1149941902:83 , srv1159350049:83 , srv1259773114:83 , srv1732411036:83 , srv1908784337:83 , srv232857239:83 , srv37498204:83 , srv909483782:83 , srv985330021:83 , srv995655787:83 , srv1222683308:84 , srv1346813548:84 , srv1661510174:84 , srv171362798:84 , srv1949815666:84 , srv239355388:84 , srv29857683:84 , srv597606703:84 , srv830180987:84 , srv947164576:84 , srv956931750:84 , srv1156475621:85 , srv117290899:85 , srv1185171544:85 , srv1751259047:85 , srv1874381902:85 , srv1886282596:85 , srv1965759903:85 , srv1984254309:85 , srv2146399080:85 , srv325321771:85 , srv918290462:85 , srv973346738:85 , srv997437701:85 , srv1293504679:86 , srv1405306101:86 , srv1480823196:86 , srv156215139:86 , srv1804950666:86 , srv1916938612:86 , srv2006612521:86 , srv2034871029:86 , srv2066008166:86 , srv391475836:86 , srv630211292:86 , srv725568652:86 , srv747977591:86 , srv81937387:86 , srv1068601747:87 , srv1083864939:87 , srv133452944:87 , srv1463006174:87 , srv1474667701:87 , srv1499456817:87 , srv1703177380:87 , srv1819961710:87 , srv1917133390:87 , srv284633849:87 , srv326813491:87 , srv346128466:87 , srv575276354:87 , srv62057421:87 , srv899079640:87 , srv988638772:87 , srv1283919546:88 , srv1472097678:88 , srv1492525003:88 , srv1498606686:88 , srv18112717:88 , srv1830635479:88 , srv1843261339:88 , srv1891922563:88 , srv2080702927:88 , srv375582984:88 , srv39387182:88 , srv434677467:88 , srv438503280:88 , srv440803995:88 , srv464234683:88 , srv505439752:88 , srv864106146:88 , srv1291197943:89 , srv1669099576:89 , srv1914302063:89 , srv1930559648:89 , srv276717301:89 , srv384392594:89 , srv751925619:89 , srv86169944:89 , srv1521614752:90 , srv154102820:90 , srv179276702:90 , srv450756119:90 , srv520997922:90 , srv981746:90 , srv1757174813:91 , srv225877013:91 , srv1590551711:93 , srv111345863:95 , srv894909235:427 }
2023-04-03 04:56:29,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-03 04:56:29,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2080702927=54, srv725568652=81, srv276717301=59, srv440803995=72, srv1498606686=21, srv1293504679=12, srv284633849=60, srv988638772=97, srv434677467=70, srv326813491=63, srv1492525003=20, srv117290899=6, srv1149941902=3, srv995655787=98, srv918290462=91, srv973346738=94, srv1283919546=10, srv747977591=82, srv1732411036=31, srv39387182=69, srv154102820=24, srv2146399080=55, srv1874381902=40, srv956931750=93, srv391475836=68, srv1156475621=4, srv156215139=25, srv29857683=61, srv997437701=99, srv171362798=30, srv81937387=84, srv1917133390=46, srv630211292=80, srv1965759903=49, srv225877013=56, srv179276702=34, srv830180987=85, srv985330021=96, srv1480823196=19, srv1521614752=23, srv909483782=90, srv864106146=87, srv1185171544=7, srv1804950666=35, srv1259773114=9, srv384392594=67, srv450756119=73, srv62057421=79, srv1463006174=16, srv1068601747=0, srv1886282596=41, srv575276354=77, srv111345863=2, srv232857239=57, srv1346813548=14, srv1830635479=38, srv1405306101=15, srv239355388=58, srv1843261339=39, srv1159350049=5, srv133452944=13, srv37498204=65, srv346128466=64, srv1499456817=22, srv1908784337=43, srv1291197943=11, srv1669099576=28, srv1472097678=17, srv1474667701=18, srv505439752=75, srv894909235=88, srv947164576=92, srv1949815666=48, srv1661510174=27, srv1751259047=32, srv1757174813=33, srv1083864939=1, srv2006612521=51, srv438503280=71, srv1590551711=26, srv86169944=86, srv899079640=89, srv520997922=76, srv1222683308=8, srv1914302063=44, srv1930559648=47, srv325321771=62, srv597606703=78, srv751925619=83, srv1703177380=29, srv2034871029=52, srv2066008166=53, srv375582984=66, srv981746=95, srv1984254309=50, srv18112717=36, srv1819961710=37, srv1891922563=42, srv1916938612=45, srv464234683=74} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-03 04:56:29,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-03 04:56:29,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-03 04:56:29,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-03 04:56:29,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-03 04:56:29,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-03 04:56:29,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-03 04:56:29,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-03 04:56:29,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-03 04:56:29,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-03 04:56:29,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-03 04:56:29,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-03 04:56:29,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-03 04:56:29,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-03 04:56:29,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-03 04:56:29,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-03 04:56:29,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-03 04:56:29,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-03 04:56:29,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-03 04:56:29,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-03 04:56:29,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-03 04:56:29,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-03 04:56:29,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-03 04:56:29,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-03 04:56:29,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-03 04:56:29,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-03 04:56:29,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-03 04:56:29,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-03 04:56:29,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-03 04:56:29,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-03 04:56:29,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-03 04:56:29,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-03 04:56:29,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-03 04:56:29,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-03 04:56:29,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-03 04:56:29,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-03 04:56:29,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-03 04:56:29,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-03 04:56:29,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-03 04:56:29,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-03 04:56:29,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-03 04:56:29,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-03 04:56:29,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-03 04:56:29,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-03 04:56:29,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-03 04:56:29,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-03 04:56:29,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-03 04:56:29,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-03 04:56:29,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-03 04:56:29,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-03 04:56:29,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-03 04:56:29,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-03 04:56:29,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-03 04:56:29,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-03 04:56:29,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-03 04:56:29,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-03 04:56:29,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-03 04:56:29,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-03 04:56:29,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-03 04:56:29,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-03 04:56:29,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-03 04:56:29,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-03 04:56:29,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-03 04:56:29,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-03 04:56:29,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-03 04:56:29,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-03 04:56:29,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-03 04:56:29,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-03 04:56:29,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-03 04:56:29,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-03 04:56:29,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-03 04:56:29,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-03 04:56:29,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-03 04:56:29,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-03 04:56:29,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-03 04:56:29,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-03 04:56:29,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-03 04:56:29,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-03 04:56:29,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-03 04:56:29,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-03 04:56:29,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-03 04:56:29,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-03 04:56:29,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-03 04:56:29,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-03 04:56:29,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-03 04:56:29,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-03 04:56:29,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-03 04:56:29,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-03 04:56:29,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-03 04:56:29,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-03 04:56:29,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-03 04:56:29,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-03 04:56:29,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-03 04:56:29,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-03 04:56:29,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-03 04:56:29,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-03 04:56:29,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-03 04:56:29,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-03 04:56:29,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-03 04:56:29,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-03 04:56:29,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-03 04:56:29,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-03 04:56:29,612 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03791221869299151, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054496985573406743, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03791221869299151, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1765, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23908333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 04:56:29,613 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-03 04:56:29,613 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18085080496521141, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03791221869299151, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054496985573406743, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03791221869299151, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1765, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23908333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-03 04:56:29,921 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 432 ms to try 5000 different iterations.  Found a solution that moves 2625 regions; Going from a computed imbalance of 0.18085080496521141 to a new imbalance of 0.013758633044720464. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02002243410688456); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031028027589130272, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2916666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02002243410688456); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01075); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 04:56:29,933 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv111345863:79 , srv1757174813:80 , srv225877013:80 , srv1590551711:81 , srv1291197943:82 , srv154102820:82 , srv1669099576:82 , srv179276702:82 , srv18112717:82 , srv1914302063:82 , srv276717301:82 , srv39387182:82 , srv464234683:82 , srv981746:82 , srv1463006174:83 , srv1843261339:83 , srv1930559648:83 , srv2080702927:83 , srv375582984:83 , srv450756119:83 , srv505439752:83 , srv520997922:83 , srv133452944:84 , srv1703177380:84 , srv1830635479:84 , srv1891922563:84 , srv384392594:84 , srv438503280:84 , srv440803995:84 , srv575276354:84 , srv62057421:84 , srv1068601747:85 , srv1283919546:85 , srv1474667701:85 , srv1498606686:85 , srv1521614752:85 , srv156215139:85 , srv1819961710:85 , srv2034871029:85 , srv346128466:85 , srv434677467:85 , srv86169944:85 , srv864106146:85 , srv899079640:85 , srv988638772:85 , srv1293504679:86 , srv1405306101:86 , srv1472097678:86 , srv1492525003:86 , srv1874381902:86 , srv1917133390:86 , srv1949815666:86 , srv2006612521:86 , srv284633849:86 , srv325321771:86 , srv326813491:86 , srv597606703:86 , srv747977591:86 , srv751925619:86 , srv81937387:86 , srv830180987:86 , srv1083864939:87 , srv1185171544:87 , srv1222683308:87 , srv1480823196:87 , srv1499456817:87 , srv1751259047:87 , srv1984254309:87 , srv2066008166:87 , srv37498204:87 , srv391475836:87 , srv630211292:87 , srv918290462:87 , srv1156475621:88 , srv117290899:88 , srv1661510174:88 , srv1804950666:88 , srv1886282596:88 , srv1908784337:88 , srv1916938612:88 , srv1965759903:88 , srv2146399080:88 , srv725568652:88 , srv909483782:88 , srv947164576:88 , srv956931750:88 , srv973346738:88 , srv997437701:88 , srv1149941902:89 , srv1346813548:89 , srv171362798:89 , srv1732411036:89 , srv232857239:89 , srv239355388:89 , srv29857683:89 , srv985330021:89 , srv1159350049:91 , srv1259773114:91 , srv995655787:91 , srv894909235:525 }
2023-04-03 04:56:29,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-03 04:56:29,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2080702927=54, srv725568652=81, srv276717301=59, srv440803995=72, srv1498606686=21, srv1293504679=12, srv284633849=60, srv988638772=97, srv434677467=70, srv326813491=63, srv1492525003=20, srv117290899=6, srv1149941902=3, srv995655787=98, srv918290462=91, srv973346738=94, srv1283919546=10, srv747977591=82, srv1732411036=31, srv39387182=69, srv154102820=24, srv2146399080=55, srv1874381902=40, srv956931750=93, srv391475836=68, srv1156475621=4, srv156215139=25, srv29857683=61, srv997437701=99, srv171362798=30, srv81937387=84, srv1917133390=46, srv630211292=80, srv1965759903=49, srv225877013=56, srv179276702=34, srv830180987=85, srv985330021=96, srv1480823196=19, srv1521614752=23, srv909483782=90, srv864106146=87, srv1185171544=7, srv1804950666=35, srv1259773114=9, srv384392594=67, srv450756119=73, srv62057421=79, srv1463006174=16, srv1068601747=0, srv1886282596=41, srv575276354=77, srv111345863=2, srv232857239=57, srv1346813548=14, srv1830635479=38, srv1405306101=15, srv239355388=58, srv1843261339=39, srv1159350049=5, srv133452944=13, srv37498204=65, srv346128466=64, srv1499456817=22, srv1908784337=43, srv1291197943=11, srv1669099576=28, srv1472097678=17, srv1474667701=18, srv505439752=75, srv894909235=88, srv947164576=92, srv1949815666=48, srv1661510174=27, srv1751259047=32, srv1757174813=33, srv1083864939=1, srv2006612521=51, srv438503280=71, srv1590551711=26, srv86169944=86, srv899079640=89, srv520997922=76, srv1222683308=8, srv1914302063=44, srv1930559648=47, srv325321771=62, srv597606703=78, srv751925619=83, srv1703177380=29, srv2034871029=52, srv2066008166=53, srv375582984=66, srv981746=95, srv1984254309=50, srv18112717=36, srv1819961710=37, srv1891922563=42, srv1916938612=45, srv464234683=74} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-03 04:56:29,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-03 04:56:29,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-03 04:56:29,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-03 04:56:29,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-03 04:56:29,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-03 04:56:29,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-03 04:56:29,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-03 04:56:29,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-03 04:56:29,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-03 04:56:29,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-03 04:56:29,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-03 04:56:29,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-03 04:56:29,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-03 04:56:29,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-03 04:56:29,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-03 04:56:29,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-03 04:56:29,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-03 04:56:29,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-03 04:56:29,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-03 04:56:29,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-03 04:56:29,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-03 04:56:29,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-03 04:56:29,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-03 04:56:29,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-03 04:56:29,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-03 04:56:29,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-03 04:56:29,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-03 04:56:29,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-03 04:56:29,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-03 04:56:29,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-03 04:56:29,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-03 04:56:29,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-03 04:56:29,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-03 04:56:29,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-03 04:56:29,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-03 04:56:29,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-03 04:56:29,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-03 04:56:29,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-03 04:56:29,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-03 04:56:29,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-03 04:56:29,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-03 04:56:29,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-03 04:56:29,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-03 04:56:29,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-03 04:56:29,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-03 04:56:29,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-03 04:56:29,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-03 04:56:29,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-03 04:56:29,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-03 04:56:29,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-03 04:56:29,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-03 04:56:29,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-03 04:56:29,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-03 04:56:29,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-03 04:56:29,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-03 04:56:29,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-03 04:56:29,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-03 04:56:29,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-03 04:56:29,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-03 04:56:29,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-03 04:56:29,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-03 04:56:29,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-03 04:56:29,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-03 04:56:29,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-03 04:56:29,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-03 04:56:29,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-03 04:56:29,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-03 04:56:29,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-03 04:56:29,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-03 04:56:29,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-03 04:56:29,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-03 04:56:29,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-03 04:56:29,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-03 04:56:29,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-03 04:56:29,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-03 04:56:29,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-03 04:56:29,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-03 04:56:29,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-03 04:56:29,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-03 04:56:29,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-03 04:56:29,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-03 04:56:29,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-03 04:56:29,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-03 04:56:29,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-03 04:56:29,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-03 04:56:29,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-03 04:56:29,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-03 04:56:29,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-03 04:56:29,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-03 04:56:29,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-03 04:56:29,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-03 04:56:29,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-03 04:56:29,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-03 04:56:29,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-03 04:56:29,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-03 04:56:29,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-03 04:56:29,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-03 04:56:29,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-03 04:56:29,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-03 04:56:29,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-03 04:56:29,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-03 04:56:30,102 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02002243410688456); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031028027589130272, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02002243410688456); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01075); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 04:56:30,102 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-03 04:56:30,102 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013740252401863107, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02002243410688456); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031028027589130272, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02002243410688456); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01075); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-03 04:56:30,425 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 436 ms to try 5000 different iterations.  Found a solution that moves 967 regions; Going from a computed imbalance of 0.013740252401863107 to a new imbalance of 6.517902581168972E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005110086268208906); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006159131805828746); MoveCostFunction : (multiplier=7.0, imbalance=0.10744444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005110086268208906); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 04:56:30,428 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1156475621:83 , srv1259773114:83 , srv1492525003:83 , srv1499456817:83 , srv1521614752:83 , srv1916938612:83 , srv520997922:83 , srv751925619:83 , srv86169944:83 , srv1159350049:84 , srv1472097678:84 , srv1590551711:84 , srv1804950666:84 , srv1886282596:84 , srv2066008166:84 , srv239355388:84 , srv284633849:84 , srv29857683:84 , srv384392594:84 , srv630211292:84 , srv725568652:84 , srv995655787:84 , srv997437701:84 , srv1068601747:85 , srv1083864939:85 , srv1149941902:85 , srv117290899:85 , srv1185171544:85 , srv1293504679:85 , srv1346813548:85 , srv1405306101:85 , srv1480823196:85 , srv1498606686:85 , srv154102820:85 , srv1661510174:85 , srv1732411036:85 , srv1830635479:85 , srv1891922563:85 , srv1917133390:85 , srv1930559648:85 , srv1965759903:85 , srv1984254309:85 , srv2006612521:85 , srv2146399080:85 , srv326813491:85 , srv346128466:85 , srv434677467:85 , srv438503280:85 , srv81937387:85 , srv864106146:85 , srv899079640:85 , srv918290462:85 , srv947164576:85 , srv973346738:85 , srv981746:85 , srv985330021:85 , srv111345863:86 , srv1222683308:86 , srv1283919546:86 , srv1291197943:86 , srv1474667701:86 , srv156215139:86 , srv1669099576:86 , srv1703177380:86 , srv171362798:86 , srv1751259047:86 , srv1819961710:86 , srv1843261339:86 , srv2034871029:86 , srv2080702927:86 , srv232857239:86 , srv325321771:86 , srv391475836:86 , srv440803995:86 , srv450756119:86 , srv747977591:86 , srv909483782:86 , srv956931750:86 , srv988638772:86 , srv133452944:87 , srv1463006174:87 , srv1757174813:87 , srv179276702:87 , srv18112717:87 , srv1874381902:87 , srv1908784337:87 , srv1914302063:87 , srv225877013:87 , srv276717301:87 , srv39387182:87 , srv464234683:87 , srv505439752:87 , srv575276354:87 , srv597606703:87 , srv62057421:87 , srv830180987:87 , srv1949815666:88 , srv37498204:88 , srv375582984:88 , srv894909235:551 }
2023-04-03 04:56:30,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-03 04:56:30,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2080702927=54, srv725568652=81, srv276717301=59, srv440803995=72, srv1498606686=21, srv1293504679=12, srv284633849=60, srv988638772=97, srv434677467=70, srv326813491=63, srv1492525003=20, srv117290899=6, srv1149941902=3, srv995655787=98, srv918290462=91, srv973346738=94, srv1283919546=10, srv747977591=82, srv1732411036=31, srv39387182=69, srv154102820=24, srv2146399080=55, srv1874381902=40, srv956931750=93, srv391475836=68, srv1156475621=4, srv156215139=25, srv29857683=61, srv997437701=99, srv171362798=30, srv81937387=84, srv1917133390=46, srv630211292=80, srv1965759903=49, srv225877013=56, srv179276702=34, srv830180987=85, srv985330021=96, srv1480823196=19, srv1521614752=23, srv909483782=90, srv864106146=87, srv1185171544=7, srv1804950666=35, srv1259773114=9, srv384392594=67, srv450756119=73, srv62057421=79, srv1463006174=16, srv1068601747=0, srv1886282596=41, srv575276354=77, srv111345863=2, srv232857239=57, srv1346813548=14, srv1830635479=38, srv1405306101=15, srv239355388=58, srv1843261339=39, srv1159350049=5, srv133452944=13, srv37498204=65, srv346128466=64, srv1499456817=22, srv1908784337=43, srv1291197943=11, srv1669099576=28, srv1472097678=17, srv1474667701=18, srv505439752=75, srv894909235=88, srv947164576=92, srv1949815666=48, srv1661510174=27, srv1751259047=32, srv1757174813=33, srv1083864939=1, srv2006612521=51, srv438503280=71, srv1590551711=26, srv86169944=86, srv899079640=89, srv520997922=76, srv1222683308=8, srv1914302063=44, srv1930559648=47, srv325321771=62, srv597606703=78, srv751925619=83, srv1703177380=29, srv2034871029=52, srv2066008166=53, srv375582984=66, srv981746=95, srv1984254309=50, srv18112717=36, srv1819961710=37, srv1891922563=42, srv1916938612=45, srv464234683=74} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-03 04:56:30,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-03 04:56:30,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-03 04:56:30,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-03 04:56:30,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-03 04:56:30,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-03 04:56:30,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-03 04:56:30,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-03 04:56:30,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-03 04:56:30,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-03 04:56:30,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-03 04:56:30,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-03 04:56:30,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-03 04:56:30,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-03 04:56:30,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-03 04:56:30,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-03 04:56:30,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-03 04:56:30,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-03 04:56:30,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-03 04:56:30,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-03 04:56:30,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-03 04:56:30,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-03 04:56:30,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-03 04:56:30,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-03 04:56:30,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-03 04:56:30,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-03 04:56:30,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-03 04:56:30,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-03 04:56:30,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-03 04:56:30,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-03 04:56:30,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-03 04:56:30,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-03 04:56:30,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-03 04:56:30,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-03 04:56:30,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-03 04:56:30,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-03 04:56:30,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-03 04:56:30,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-03 04:56:30,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-03 04:56:30,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-03 04:56:30,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-03 04:56:30,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-03 04:56:30,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-03 04:56:30,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-03 04:56:30,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-03 04:56:30,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-03 04:56:30,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-03 04:56:30,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-03 04:56:30,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-03 04:56:30,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-03 04:56:30,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-03 04:56:30,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-03 04:56:30,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-03 04:56:30,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-03 04:56:30,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-03 04:56:30,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-03 04:56:30,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-03 04:56:30,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-03 04:56:30,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-03 04:56:30,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-03 04:56:30,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-03 04:56:30,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-03 04:56:30,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-03 04:56:30,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-03 04:56:30,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-03 04:56:30,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-03 04:56:30,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-03 04:56:30,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-03 04:56:30,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-03 04:56:30,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-03 04:56:30,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-03 04:56:30,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-03 04:56:30,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-03 04:56:30,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-03 04:56:30,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-03 04:56:30,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-03 04:56:30,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-03 04:56:30,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-03 04:56:30,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-03 04:56:30,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-03 04:56:30,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-03 04:56:30,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-03 04:56:30,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-03 04:56:30,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-03 04:56:30,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-03 04:56:30,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-03 04:56:30,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-03 04:56:30,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-03 04:56:30,470 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-03 04:56:30,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-03 04:56:30,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-03 04:56:30,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-03 04:56:30,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-03 04:56:30,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-03 04:56:30,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-03 04:56:30,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-03 04:56:30,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-03 04:56:30,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-03 04:56:30,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-03 04:56:30,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-03 04:56:30,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-03 04:56:30,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-03 04:56:30,629 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005110086268208906); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006159131805828746); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005110086268208906); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 04:56:30,630 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-03 04:56:30,630 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.450191793957296E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005110086268208906); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006159131805828746); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005110086268208906); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-03 04:56:30,919 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 423 ms to try 5000 different iterations.  Found a solution that moves 258 regions; Going from a computed imbalance of 6.450191793957296E-4 to a new imbalance of 2.0685594857454618E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.909066927536782E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.028666666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.909066927536782E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 04:56:30,920 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv391475836:82 , srv1083864939:83 , srv111345863:83 , srv1283919546:83 , srv171362798:83 , srv1751259047:83 , srv1819961710:83 , srv1874381902:83 , srv1914302063:83 , srv2146399080:83 , srv375582984:83 , srv440803995:83 , srv450756119:83 , srv1159350049:84 , srv117290899:84 , srv133452944:84 , srv1346813548:84 , srv1463006174:84 , srv1472097678:84 , srv1474667701:84 , srv1480823196:84 , srv1492525003:84 , srv1498606686:84 , srv1521614752:84 , srv154102820:84 , srv1590551711:84 , srv1669099576:84 , srv1757174813:84 , srv179276702:84 , srv1917133390:84 , srv1949815666:84 , srv1965759903:84 , srv225877013:84 , srv232857239:84 , srv276717301:84 , srv326813491:84 , srv346128466:84 , srv37498204:84 , srv434677467:84 , srv575276354:84 , srv62057421:84 , srv725568652:84 , srv747977591:84 , srv751925619:84 , srv864106146:84 , srv956931750:84 , srv973346738:84 , srv1068601747:85 , srv1149941902:85 , srv1156475621:85 , srv1185171544:85 , srv1222683308:85 , srv1259773114:85 , srv1293504679:85 , srv1405306101:85 , srv156215139:85 , srv1661510174:85 , srv1703177380:85 , srv1732411036:85 , srv1804950666:85 , srv18112717:85 , srv1830635479:85 , srv1843261339:85 , srv1891922563:85 , srv1908784337:85 , srv1916938612:85 , srv1930559648:85 , srv1984254309:85 , srv2034871029:85 , srv2066008166:85 , srv2080702927:85 , srv239355388:85 , srv284633849:85 , srv325321771:85 , srv384392594:85 , srv39387182:85 , srv438503280:85 , srv505439752:85 , srv520997922:85 , srv630211292:85 , srv81937387:85 , srv830180987:85 , srv86169944:85 , srv899079640:85 , srv909483782:85 , srv918290462:85 , srv947164576:85 , srv981746:85 , srv985330021:85 , srv988638772:85 , srv995655787:85 , srv997437701:85 , srv1291197943:86 , srv1499456817:86 , srv1886282596:86 , srv2006612521:86 , srv29857683:86 , srv464234683:86 , srv597606703:86 , srv894909235:639 }
2023-04-03 04:56:30,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-03 04:56:30,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2080702927=54, srv725568652=81, srv276717301=59, srv440803995=72, srv1498606686=21, srv1293504679=12, srv284633849=60, srv988638772=97, srv434677467=70, srv326813491=63, srv1492525003=20, srv117290899=6, srv1149941902=3, srv995655787=98, srv918290462=91, srv973346738=94, srv1283919546=10, srv747977591=82, srv1732411036=31, srv39387182=69, srv154102820=24, srv2146399080=55, srv1874381902=40, srv956931750=93, srv391475836=68, srv1156475621=4, srv156215139=25, srv29857683=61, srv997437701=99, srv171362798=30, srv81937387=84, srv1917133390=46, srv630211292=80, srv1965759903=49, srv225877013=56, srv179276702=34, srv830180987=85, srv985330021=96, srv1480823196=19, srv1521614752=23, srv909483782=90, srv864106146=87, srv1185171544=7, srv1804950666=35, srv1259773114=9, srv384392594=67, srv450756119=73, srv62057421=79, srv1463006174=16, srv1068601747=0, srv1886282596=41, srv575276354=77, srv111345863=2, srv232857239=57, srv1346813548=14, srv1830635479=38, srv1405306101=15, srv239355388=58, srv1843261339=39, srv1159350049=5, srv133452944=13, srv37498204=65, srv346128466=64, srv1499456817=22, srv1908784337=43, srv1291197943=11, srv1669099576=28, srv1472097678=17, srv1474667701=18, srv505439752=75, srv894909235=88, srv947164576=92, srv1949815666=48, srv1661510174=27, srv1751259047=32, srv1757174813=33, srv1083864939=1, srv2006612521=51, srv438503280=71, srv1590551711=26, srv86169944=86, srv899079640=89, srv520997922=76, srv1222683308=8, srv1914302063=44, srv1930559648=47, srv325321771=62, srv597606703=78, srv751925619=83, srv1703177380=29, srv2034871029=52, srv2066008166=53, srv375582984=66, srv981746=95, srv1984254309=50, srv18112717=36, srv1819961710=37, srv1891922563=42, srv1916938612=45, srv464234683=74} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-03 04:56:30,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-03 04:56:30,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-03 04:56:30,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-03 04:56:30,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-03 04:56:30,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-03 04:56:30,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-03 04:56:30,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-03 04:56:30,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-03 04:56:30,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-03 04:56:30,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-03 04:56:30,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-03 04:56:30,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-03 04:56:30,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-03 04:56:30,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-03 04:56:30,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-03 04:56:30,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-03 04:56:30,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-03 04:56:30,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-03 04:56:30,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-03 04:56:30,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-03 04:56:30,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-03 04:56:30,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-03 04:56:30,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-03 04:56:30,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-03 04:56:30,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-03 04:56:30,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-03 04:56:30,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-03 04:56:30,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-03 04:56:30,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-03 04:56:30,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-03 04:56:30,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-03 04:56:30,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-03 04:56:30,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-03 04:56:30,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-03 04:56:30,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-03 04:56:30,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-03 04:56:30,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-03 04:56:30,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-03 04:56:30,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-03 04:56:30,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-03 04:56:30,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-03 04:56:30,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-03 04:56:30,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-03 04:56:30,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-03 04:56:30,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-03 04:56:30,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-03 04:56:30,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-03 04:56:30,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-03 04:56:30,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-03 04:56:30,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-03 04:56:30,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-03 04:56:30,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-03 04:56:30,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-03 04:56:30,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-03 04:56:30,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-03 04:56:30,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-03 04:56:30,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-03 04:56:30,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-03 04:56:30,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-03 04:56:30,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-03 04:56:30,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-03 04:56:30,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-03 04:56:30,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-03 04:56:30,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-03 04:56:30,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-03 04:56:30,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-03 04:56:30,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-03 04:56:30,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-03 04:56:30,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-03 04:56:30,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-03 04:56:30,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-03 04:56:30,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-03 04:56:30,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-03 04:56:30,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-03 04:56:30,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-03 04:56:30,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-03 04:56:30,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-03 04:56:30,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-03 04:56:30,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-03 04:56:30,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-03 04:56:30,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-03 04:56:30,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-03 04:56:30,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-03 04:56:30,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-03 04:56:30,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-03 04:56:30,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-03 04:56:30,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-03 04:56:30,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-03 04:56:30,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-03 04:56:30,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-03 04:56:30,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-03 04:56:30,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-03 04:56:30,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-03 04:56:30,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-03 04:56:30,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-03 04:56:30,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-03 04:56:30,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-03 04:56:30,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-03 04:56:30,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-03 04:56:30,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-03 04:56:30,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-03 04:56:31,064 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.887904024518865E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.909066927536782E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.909066927536782E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 04:56:31,064 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv391475836:82 , srv1083864939:83 , srv111345863:83 , srv1283919546:83 , srv171362798:83 , srv1751259047:83 , srv1819961710:83 , srv1874381902:83 , srv1914302063:83 , srv2146399080:83 , srv375582984:83 , srv440803995:83 , srv450756119:83 , srv1159350049:84 , srv117290899:84 , srv133452944:84 , srv1346813548:84 , srv1463006174:84 , srv1472097678:84 , srv1474667701:84 , srv1480823196:84 , srv1492525003:84 , srv1498606686:84 , srv1521614752:84 , srv154102820:84 , srv1590551711:84 , srv1669099576:84 , srv1757174813:84 , srv179276702:84 , srv1917133390:84 , srv1949815666:84 , srv1965759903:84 , srv225877013:84 , srv232857239:84 , srv276717301:84 , srv326813491:84 , srv346128466:84 , srv37498204:84 , srv434677467:84 , srv575276354:84 , srv62057421:84 , srv725568652:84 , srv747977591:84 , srv751925619:84 , srv864106146:84 , srv956931750:84 , srv973346738:84 , srv1068601747:85 , srv1149941902:85 , srv1156475621:85 , srv1185171544:85 , srv1222683308:85 , srv1259773114:85 , srv1293504679:85 , srv1405306101:85 , srv156215139:85 , srv1661510174:85 , srv1703177380:85 , srv1732411036:85 , srv1804950666:85 , srv18112717:85 , srv1830635479:85 , srv1843261339:85 , srv1891922563:85 , srv1908784337:85 , srv1916938612:85 , srv1930559648:85 , srv1984254309:85 , srv2034871029:85 , srv2066008166:85 , srv2080702927:85 , srv239355388:85 , srv284633849:85 , srv325321771:85 , srv384392594:85 , srv39387182:85 , srv438503280:85 , srv505439752:85 , srv520997922:85 , srv630211292:85 , srv81937387:85 , srv830180987:85 , srv86169944:85 , srv899079640:85 , srv909483782:85 , srv918290462:85 , srv947164576:85 , srv981746:85 , srv985330021:85 , srv988638772:85 , srv995655787:85 , srv997437701:85 , srv1291197943:86 , srv1499456817:86 , srv1886282596:86 , srv2006612521:86 , srv29857683:86 , srv464234683:86 , srv597606703:86 , srv894909235:639 }
2023-04-03 04:56:31,092 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=442 (was 442), ProcessCount=169 (was 169), AvailableMemoryMB=4182 (was 4681)
2023-04-03 04:56:31,102 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=442, ProcessCount=169, AvailableMemoryMB=4182
2023-04-03 04:56:31,103 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-03 04:56:31,103 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-04-03 04:56:31,115 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1127959038:84 , srv117190644:84 , srv1250780637:84 , srv1289523388:84 , srv134092621:84 , srv1347211160:84 , srv1352697616:84 , srv141291169:84 , srv1569057496:84 , srv1658215056:84 , srv1686034335:84 , srv181776300:84 , srv1846353268:84 , srv1902977307:84 , srv2034960625:84 , srv2093700624:84 , srv224467333:84 , srv276390332:84 , srv368314460:84 , srv414317596:84 , srv437877098:84 , srv454140492:84 , srv458209745:84 , srv54449838:84 , srv713611387:84 , srv762127682:84 , srv772577786:84 , srv827274270:84 , srv860301273:84 , srv1832075685:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-04-03 04:56:31,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-03 04:56:31,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1658215056=9, srv713611387=25, srv1289523388=3, srv54449838=24, srv276390332=18, srv414317596=20, srv437877098=21, srv117190644=1, srv860301273=29, srv2093700624=16, srv1569057496=8, srv181776300=11, srv1347211160=5, srv827274270=28, srv134092621=4, srv1250780637=2, srv1902977307=14, srv1846353268=13, srv454140492=22, srv1832075685=12, srv772577786=27, srv458209745=23, srv1127959038=0, srv224467333=17, srv141291169=7, srv1352697616=6, srv762127682=26, srv368314460=19, srv1686034335=10, srv2034960625=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-03 04:56:31,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-03 04:56:31,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-03 04:56:31,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-03 04:56:31,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-03 04:56:31,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-03 04:56:31,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-03 04:56:31,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-03 04:56:31,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-03 04:56:31,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-03 04:56:31,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-03 04:56:31,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-03 04:56:31,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-03 04:56:31,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-03 04:56:31,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-03 04:56:31,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-03 04:56:31,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-03 04:56:31,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-03 04:56:31,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-03 04:56:31,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-03 04:56:31,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-03 04:56:31,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-03 04:56:31,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-03 04:56:31,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-03 04:56:31,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-03 04:56:31,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-03 04:56:31,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-03 04:56:31,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-03 04:56:31,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-03 04:56:31,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-03 04:56:31,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-03 04:56:31,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-03 04:56:31,152 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1321399620052621, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 04:56:31,152 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912609411969792, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1321399620052621, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-04-03 04:58:31,138 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6035409 different iterations.  Found a solution that moves 1849 regions; Going from a computed imbalance of 0.9912609411969792 to a new imbalance of 4.3156582404131396E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6848148148148148, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 04:58:31,144 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1127959038:90 , srv117190644:90 , srv1250780637:90 , srv1289523388:90 , srv134092621:90 , srv1347211160:90 , srv1352697616:90 , srv141291169:90 , srv1569057496:90 , srv1658215056:90 , srv1686034335:90 , srv181776300:90 , srv1832075685:90 , srv1846353268:90 , srv1902977307:90 , srv2034960625:90 , srv2093700624:90 , srv224467333:90 , srv276390332:90 , srv368314460:90 , srv414317596:90 , srv437877098:90 , srv454140492:90 , srv458209745:90 , srv54449838:90 , srv713611387:90 , srv762127682:90 , srv772577786:90 , srv827274270:90 , srv860301273:90 }
2023-04-03 04:58:31,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-03 04:58:31,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1658215056=9, srv713611387=25, srv1289523388=3, srv54449838=24, srv276390332=18, srv414317596=20, srv437877098=21, srv117190644=1, srv860301273=29, srv2093700624=16, srv1569057496=8, srv181776300=11, srv1347211160=5, srv827274270=28, srv134092621=4, srv1250780637=2, srv1902977307=14, srv1846353268=13, srv454140492=22, srv1832075685=12, srv772577786=27, srv458209745=23, srv1127959038=0, srv224467333=17, srv141291169=7, srv1352697616=6, srv762127682=26, srv368314460=19, srv1686034335=10, srv2034960625=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-03 04:58:31,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-03 04:58:31,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-03 04:58:31,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-03 04:58:31,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-03 04:58:31,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-03 04:58:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-03 04:58:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-03 04:58:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-03 04:58:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-03 04:58:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-03 04:58:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-03 04:58:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-03 04:58:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-03 04:58:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-03 04:58:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-03 04:58:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-03 04:58:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-03 04:58:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-03 04:58:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-03 04:58:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-03 04:58:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-03 04:58:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-03 04:58:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-03 04:58:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-03 04:58:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-03 04:58:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-03 04:58:31,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-03 04:58:31,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-03 04:58:31,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-03 04:58:31,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-03 04:58:31,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-03 04:58:31,163 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-03 04:58:31,164 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1127959038:90 , srv117190644:90 , srv1250780637:90 , srv1289523388:90 , srv134092621:90 , srv1347211160:90 , srv1352697616:90 , srv141291169:90 , srv1569057496:90 , srv1658215056:90 , srv1686034335:90 , srv181776300:90 , srv1832075685:90 , srv1846353268:90 , srv1902977307:90 , srv2034960625:90 , srv2093700624:90 , srv224467333:90 , srv276390332:90 , srv368314460:90 , srv414317596:90 , srv437877098:90 , srv454140492:90 , srv458209745:90 , srv54449838:90 , srv713611387:90 , srv762127682:90 , srv772577786:90 , srv827274270:90 , srv860301273:90 }
2023-04-03 04:58:31,179 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=231 (was 442), ProcessCount=169 (was 169), AvailableMemoryMB=4407 (was 4182) - AvailableMemoryMB LEAK? -