2023-04-08 21:01:39,422 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-04-08 21:01:39,701 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-04-08 21:01:39,802 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-04-08 21:01:39,854 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-08 21:01:39,860 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-08 21:01:39,870 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=159, ProcessCount=165, AvailableMemoryMB=4922
2023-04-08 21:01:39,872 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-08 21:01:39,873 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-08 21:01:39,997 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1039636446:84 , srv106482882:84 , srv1127694968:84 , srv1142695142:84 , srv1171505731:84 , srv1176324266:84 , srv1197400377:84 , srv1199002850:84 , srv1218014158:84 , srv1221466471:84 , srv122978775:84 , srv1247520683:84 , srv1270699725:84 , srv1312946646:84 , srv1322078838:84 , srv1340852699:84 , srv1378038673:84 , srv1389655910:84 , srv1392296251:84 , srv139556414:84 , srv1399479203:84 , srv1408086951:84 , srv1418536510:84 , srv1439648029:84 , srv1470917578:84 , srv1521769290:84 , srv1556745260:84 , srv1558363389:84 , srv1589728830:84 , srv1606346206:84 , srv1623022434:84 , srv1643591997:84 , srv1730610746:84 , srv1744571159:84 , srv1753064799:84 , srv1757502019:84 , srv1762696074:84 , srv1767083400:84 , srv1768560338:84 , srv1844984463:84 , srv1916375045:84 , srv1940322592:84 , srv1959710238:84 , srv1969739994:84 , srv1975344943:84 , srv1979171709:84 , srv1993378048:84 , srv1995523839:84 , srv2017989594:84 , srv2036065400:84 , srv2078603354:84 , srv2092490940:84 , srv2108467319:84 , srv2131413470:84 , srv2142520698:84 , srv219562644:84 , srv268008008:84 , srv270580743:84 , srv273491036:84 , srv311089809:84 , srv35288526:84 , srv354622703:84 , srv35683316:84 , srv392842158:84 , srv401534877:84 , srv426494307:84 , srv431057273:84 , srv46669828:84 , srv471392759:84 , srv479625257:84 , srv499539334:84 , srv500943747:84 , srv522983751:84 , srv529655867:84 , srv536633585:84 , srv56476101:84 , srv564833147:84 , srv565986631:84 , srv571732148:84 , srv571858677:84 , srv584649307:84 , srv589151788:84 , srv599491235:84 , srv601459500:84 , srv62241484:84 , srv647135200:84 , srv702614480:84 , srv703581742:84 , srv703660695:84 , srv745188519:84 , srv762121503:84 , srv775834625:84 , srv780059868:84 , srv819111366:84 , srv882947513:84 , srv924199915:84 , srv974382741:84 , srv98145949:84 , srv993207587:84 , srv1928295909:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-04-08 21:01:40,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-08 21:01:40,012 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv702614480=87, srv924199915=96, srv1408086951=21, srv1928295909=41, srv745188519=90, srv2131413470=54, srv1197400377=6, srv1399479203=20, srv1993378048=47, srv1142695142=3, srv270580743=58, srv529655867=74, srv565986631=78, srv1940322592=42, srv1606346206=29, srv35288526=61, srv1270699725=12, srv1556745260=26, srv392842158=64, srv139556414=19, srv780059868=93, srv479625257=70, srv2108467319=53, srv431057273=67, srv1470917578=24, srv1340852699=15, srv703660695=89, srv1521769290=25, srv268008008=57, srv647135200=86, srv1039636446=0, srv1199002850=7, srv1218014158=8, srv1221466471=9, srv564833147=77, srv1389655910=17, srv882947513=95, srv2078603354=51, srv2017989594=49, srv122978775=10, srv1171505731=4, srv401534877=65, srv1392296251=18, srv1247520683=11, srv1744571159=33, srv1127694968=2, srv1623022434=30, srv536633585=75, srv599491235=83, srv1589728830=28, srv601459500=84, srv2092490940=52, srv2036065400=50, srv1322078838=14, srv775834625=92, srv571732148=79, srv1762696074=36, srv219562644=56, srv1979171709=46, srv993207587=99, srv471392759=69, srv1643591997=31, srv1558363389=27, srv98145949=98, srv106482882=1, srv1439648029=23, srv273491036=59, srv974382741=97, srv584649307=81, srv1767083400=37, srv762121503=91, srv354622703=62, srv426494307=66, srv1176324266=5, srv311089809=60, srv1312946646=13, srv1969739994=44, srv703581742=88, srv56476101=76, srv1975344943=45, srv1418536510=22, srv46669828=68, srv1844984463=39, srv571858677=80, srv1730610746=32, srv1757502019=35, srv2142520698=55, srv500943747=72, srv1995523839=48, srv589151788=82, srv1768560338=38, srv62241484=85, srv522983751=73, srv819111366=94, srv499539334=71, srv35683316=63, srv1378038673=16, srv1959710238=43, srv1916375045=40, srv1753064799=34} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-08 21:01:40,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-08 21:01:40,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-08 21:01:40,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-08 21:01:40,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-08 21:01:40,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-08 21:01:40,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-08 21:01:40,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-08 21:01:40,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-08 21:01:40,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-08 21:01:40,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-08 21:01:40,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-08 21:01:40,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-08 21:01:40,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-08 21:01:40,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-08 21:01:40,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-08 21:01:40,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-08 21:01:40,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-08 21:01:40,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-08 21:01:40,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-08 21:01:40,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-08 21:01:40,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-08 21:01:40,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-08 21:01:40,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-08 21:01:40,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-08 21:01:40,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-08 21:01:40,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-08 21:01:40,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-08 21:01:40,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-08 21:01:40,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-08 21:01:40,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-08 21:01:40,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-08 21:01:40,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-08 21:01:40,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-08 21:01:40,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-08 21:01:40,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-08 21:01:40,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-08 21:01:40,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-08 21:01:40,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-08 21:01:40,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-08 21:01:40,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-08 21:01:40,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-08 21:01:40,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-08 21:01:40,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-08 21:01:40,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-08 21:01:40,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-08 21:01:40,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-08 21:01:40,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-08 21:01:40,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-08 21:01:40,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-08 21:01:40,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-08 21:01:40,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-08 21:01:40,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-08 21:01:40,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-08 21:01:40,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-08 21:01:40,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-08 21:01:40,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-08 21:01:40,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-08 21:01:40,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-08 21:01:40,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-08 21:01:40,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-08 21:01:40,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-08 21:01:40,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-08 21:01:40,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-08 21:01:40,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-08 21:01:40,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-08 21:01:40,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-08 21:01:40,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-08 21:01:40,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-08 21:01:40,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-08 21:01:40,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-08 21:01:40,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-08 21:01:40,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-08 21:01:40,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-08 21:01:40,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-08 21:01:40,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-08 21:01:40,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-08 21:01:40,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-08 21:01:40,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-08 21:01:40,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-08 21:01:40,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-08 21:01:40,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-08 21:01:40,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-08 21:01:40,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-08 21:01:40,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-08 21:01:40,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-08 21:01:40,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-08 21:01:40,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-08 21:01:40,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-08 21:01:40,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-08 21:01:40,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-08 21:01:40,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-08 21:01:40,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-08 21:01:40,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-08 21:01:40,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-08 21:01:40,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-08 21:01:40,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-08 21:01:40,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-08 21:01:40,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-08 21:01:40,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-08 21:01:40,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-08 21:01:40,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-08 21:01:40,347 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 21:01:40,348 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-08 21:01:40,348 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-08 21:01:40,715 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 598 ms to try 5000 different iterations.  Found a solution that moves 4716 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17904363992232467. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039836929102457855, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056965935831142504, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.524, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039836929102457855, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17408333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24258333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 21:01:40,728 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1979171709:81 , srv1556745260:82 , srv1730610746:82 , srv703660695:82 , srv1039636446:83 , srv1127694968:83 , srv1197400377:83 , srv1312946646:83 , srv1418536510:83 , srv1768560338:83 , srv2092490940:83 , srv426494307:83 , srv536633585:83 , srv98145949:83 , srv1218014158:84 , srv1521769290:84 , srv1916375045:84 , srv1993378048:84 , srv1995523839:84 , srv565986631:84 , srv589151788:84 , srv601459500:84 , srv702614480:84 , srv745188519:84 , srv819111366:84 , srv1199002850:85 , srv122978775:85 , srv1270699725:85 , srv1439648029:85 , srv1623022434:85 , srv1753064799:85 , srv1757502019:85 , srv2108467319:85 , srv268008008:85 , srv35288526:85 , srv392842158:85 , srv479625257:85 , srv500943747:85 , srv780059868:85 , srv974382741:85 , srv106482882:86 , srv1392296251:86 , srv139556414:86 , srv1767083400:86 , srv1959710238:86 , srv2131413470:86 , srv219562644:86 , srv270580743:86 , srv311089809:86 , srv499539334:86 , srv522983751:86 , srv529655867:86 , srv564833147:86 , srv775834625:86 , srv1408086951:87 , srv1558363389:87 , srv1589728830:87 , srv1762696074:87 , srv1844984463:87 , srv1975344943:87 , srv2017989594:87 , srv2142520698:87 , srv273491036:87 , srv354622703:87 , srv401534877:87 , srv647135200:87 , srv1378038673:88 , srv1606346206:88 , srv1969739994:88 , srv2036065400:88 , srv35683316:88 , srv46669828:88 , srv471392759:88 , srv571732148:88 , srv62241484:88 , srv703581742:88 , srv762121503:88 , srv1142695142:89 , srv1221466471:89 , srv1247520683:89 , srv1340852699:89 , srv1470917578:89 , srv1643591997:89 , srv2078603354:89 , srv584649307:89 , srv1322078838:90 , srv1389655910:90 , srv1399479203:90 , srv1940322592:90 , srv431057273:90 , srv56476101:90 , srv882947513:90 , srv924199915:90 , srv1744571159:91 , srv1171505731:92 , srv1176324266:92 , srv599491235:92 , srv993207587:92 , srv571858677:93 , srv1928295909:444 }
2023-04-08 21:01:40,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-08 21:01:40,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv702614480=87, srv924199915=96, srv1408086951=21, srv1928295909=41, srv745188519=90, srv2131413470=54, srv1197400377=6, srv1399479203=20, srv1993378048=47, srv1142695142=3, srv270580743=58, srv529655867=74, srv565986631=78, srv1940322592=42, srv1606346206=29, srv35288526=61, srv1270699725=12, srv1556745260=26, srv392842158=64, srv139556414=19, srv780059868=93, srv479625257=70, srv2108467319=53, srv431057273=67, srv1470917578=24, srv1340852699=15, srv703660695=89, srv1521769290=25, srv268008008=57, srv647135200=86, srv1039636446=0, srv1199002850=7, srv1218014158=8, srv1221466471=9, srv564833147=77, srv1389655910=17, srv882947513=95, srv2078603354=51, srv2017989594=49, srv122978775=10, srv1171505731=4, srv401534877=65, srv1392296251=18, srv1247520683=11, srv1744571159=33, srv1127694968=2, srv1623022434=30, srv536633585=75, srv599491235=83, srv1589728830=28, srv601459500=84, srv2092490940=52, srv2036065400=50, srv1322078838=14, srv775834625=92, srv571732148=79, srv1762696074=36, srv219562644=56, srv1979171709=46, srv993207587=99, srv471392759=69, srv1643591997=31, srv1558363389=27, srv98145949=98, srv106482882=1, srv1439648029=23, srv273491036=59, srv974382741=97, srv584649307=81, srv1767083400=37, srv762121503=91, srv354622703=62, srv426494307=66, srv1176324266=5, srv311089809=60, srv1312946646=13, srv1969739994=44, srv703581742=88, srv56476101=76, srv1975344943=45, srv1418536510=22, srv46669828=68, srv1844984463=39, srv571858677=80, srv1730610746=32, srv1757502019=35, srv2142520698=55, srv500943747=72, srv1995523839=48, srv589151788=82, srv1768560338=38, srv62241484=85, srv522983751=73, srv819111366=94, srv499539334=71, srv35683316=63, srv1378038673=16, srv1959710238=43, srv1916375045=40, srv1753064799=34} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-08 21:01:40,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-08 21:01:40,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-08 21:01:40,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-08 21:01:40,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-08 21:01:40,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-08 21:01:40,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-08 21:01:40,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-08 21:01:40,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-08 21:01:40,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-08 21:01:40,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-08 21:01:40,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-08 21:01:40,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-08 21:01:40,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-08 21:01:40,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-08 21:01:40,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-08 21:01:40,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-08 21:01:40,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-08 21:01:40,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-08 21:01:40,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-08 21:01:40,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-08 21:01:40,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-08 21:01:40,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-08 21:01:40,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-08 21:01:40,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-08 21:01:40,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-08 21:01:40,749 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-08 21:01:40,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-08 21:01:40,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-08 21:01:40,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-08 21:01:40,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-08 21:01:40,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-08 21:01:40,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-08 21:01:40,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-08 21:01:40,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-08 21:01:40,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-08 21:01:40,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-08 21:01:40,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-08 21:01:40,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-08 21:01:40,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-08 21:01:40,750 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-08 21:01:40,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-08 21:01:40,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-08 21:01:40,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-08 21:01:40,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-08 21:01:40,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-08 21:01:40,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-08 21:01:40,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-08 21:01:40,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-08 21:01:40,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-08 21:01:40,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-08 21:01:40,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-08 21:01:40,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-08 21:01:40,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-08 21:01:40,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-08 21:01:40,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-08 21:01:40,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-08 21:01:40,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-08 21:01:40,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-08 21:01:40,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-08 21:01:40,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-08 21:01:40,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-08 21:01:40,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-08 21:01:40,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-08 21:01:40,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-08 21:01:40,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-08 21:01:40,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-08 21:01:40,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-08 21:01:40,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-08 21:01:40,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-08 21:01:40,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-08 21:01:40,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-08 21:01:40,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-08 21:01:40,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-08 21:01:40,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-08 21:01:40,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-08 21:01:40,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-08 21:01:40,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-08 21:01:40,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-08 21:01:40,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-08 21:01:40,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-08 21:01:40,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-08 21:01:40,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-08 21:01:40,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-08 21:01:40,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-08 21:01:40,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-08 21:01:40,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-08 21:01:40,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-08 21:01:40,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-08 21:01:40,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-08 21:01:40,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-08 21:01:40,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-08 21:01:40,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-08 21:01:40,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-08 21:01:40,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-08 21:01:40,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-08 21:01:40,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-08 21:01:40,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-08 21:01:40,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-08 21:01:40,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-08 21:01:40,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-08 21:01:40,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-08 21:01:40,855 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039836929102457855, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056965935831142504, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039836929102457855, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17408333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24258333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 21:01:40,855 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-08 21:01:40,855 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17901061778452834, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039836929102457855, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056965935831142504, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039836929102457855, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17408333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24258333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-08 21:01:41,062 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 290 ms to try 5000 different iterations.  Found a solution that moves 2611 regions; Going from a computed imbalance of 0.17901061778452834 to a new imbalance of 0.013909170081806192. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022049270859364138); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03399016164516465, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2901111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022049270859364138); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 21:01:41,071 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv571858677:78 , srv56476101:80 , srv599491235:80 , srv1221466471:81 , srv1247520683:81 , srv1322078838:81 , srv1744571159:81 , srv431057273:81 , srv584649307:81 , srv993207587:81 , srv1399479203:82 , srv1171505731:83 , srv1176324266:83 , srv1340852699:83 , srv1389655910:83 , srv1969739994:83 , srv2036065400:83 , srv571732148:83 , srv647135200:83 , srv882947513:83 , srv924199915:83 , srv1142695142:84 , srv1589728830:84 , srv1643591997:84 , srv1940322592:84 , srv1959710238:84 , srv2078603354:84 , srv2131413470:84 , srv35683316:84 , srv46669828:84 , srv499539334:84 , srv62241484:84 , srv122978775:85 , srv1378038673:85 , srv1392296251:85 , srv1408086951:85 , srv1470917578:85 , srv1606346206:85 , srv1753064799:85 , srv1767083400:85 , srv2017989594:85 , srv273491036:85 , srv354622703:85 , srv401534877:85 , srv471392759:85 , srv529655867:85 , srv703581742:85 , srv762121503:85 , srv1039636446:86 , srv106482882:86 , srv1199002850:86 , srv1418536510:86 , srv1558363389:86 , srv1762696074:86 , srv1975344943:86 , srv2142520698:86 , srv219562644:86 , srv268008008:86 , srv270580743:86 , srv564833147:86 , srv775834625:86 , srv780059868:86 , srv974382741:86 , srv139556414:87 , srv1844984463:87 , srv311089809:87 , srv35288526:87 , srv426494307:87 , srv500943747:87 , srv522983751:87 , srv702614480:87 , srv1218014158:88 , srv1270699725:88 , srv1521769290:88 , srv1623022434:88 , srv1757502019:88 , srv1916375045:88 , srv1993378048:88 , srv1995523839:88 , srv2108467319:88 , srv479625257:88 , srv536633585:88 , srv565986631:88 , srv703660695:88 , srv745188519:88 , srv819111366:88 , srv1197400377:89 , srv1439648029:89 , srv1556745260:89 , srv1768560338:89 , srv1979171709:89 , srv589151788:89 , srv98145949:89 , srv1127694968:90 , srv1730610746:90 , srv2092490940:90 , srv392842158:91 , srv601459500:91 , srv1312946646:92 , srv1928295909:526 }
2023-04-08 21:01:41,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-08 21:01:41,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv702614480=87, srv924199915=96, srv1408086951=21, srv1928295909=41, srv745188519=90, srv2131413470=54, srv1197400377=6, srv1399479203=20, srv1993378048=47, srv1142695142=3, srv270580743=58, srv529655867=74, srv565986631=78, srv1940322592=42, srv1606346206=29, srv35288526=61, srv1270699725=12, srv1556745260=26, srv392842158=64, srv139556414=19, srv780059868=93, srv479625257=70, srv2108467319=53, srv431057273=67, srv1470917578=24, srv1340852699=15, srv703660695=89, srv1521769290=25, srv268008008=57, srv647135200=86, srv1039636446=0, srv1199002850=7, srv1218014158=8, srv1221466471=9, srv564833147=77, srv1389655910=17, srv882947513=95, srv2078603354=51, srv2017989594=49, srv122978775=10, srv1171505731=4, srv401534877=65, srv1392296251=18, srv1247520683=11, srv1744571159=33, srv1127694968=2, srv1623022434=30, srv536633585=75, srv599491235=83, srv1589728830=28, srv601459500=84, srv2092490940=52, srv2036065400=50, srv1322078838=14, srv775834625=92, srv571732148=79, srv1762696074=36, srv219562644=56, srv1979171709=46, srv993207587=99, srv471392759=69, srv1643591997=31, srv1558363389=27, srv98145949=98, srv106482882=1, srv1439648029=23, srv273491036=59, srv974382741=97, srv584649307=81, srv1767083400=37, srv762121503=91, srv354622703=62, srv426494307=66, srv1176324266=5, srv311089809=60, srv1312946646=13, srv1969739994=44, srv703581742=88, srv56476101=76, srv1975344943=45, srv1418536510=22, srv46669828=68, srv1844984463=39, srv571858677=80, srv1730610746=32, srv1757502019=35, srv2142520698=55, srv500943747=72, srv1995523839=48, srv589151788=82, srv1768560338=38, srv62241484=85, srv522983751=73, srv819111366=94, srv499539334=71, srv35683316=63, srv1378038673=16, srv1959710238=43, srv1916375045=40, srv1753064799=34} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-08 21:01:41,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-08 21:01:41,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-08 21:01:41,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-08 21:01:41,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-08 21:01:41,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-08 21:01:41,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-08 21:01:41,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-08 21:01:41,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-08 21:01:41,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-08 21:01:41,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-08 21:01:41,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-08 21:01:41,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-08 21:01:41,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-08 21:01:41,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-08 21:01:41,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-08 21:01:41,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-08 21:01:41,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-08 21:01:41,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-08 21:01:41,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-08 21:01:41,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-08 21:01:41,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-08 21:01:41,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-08 21:01:41,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-08 21:01:41,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-08 21:01:41,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-08 21:01:41,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-08 21:01:41,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-08 21:01:41,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-08 21:01:41,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-08 21:01:41,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-08 21:01:41,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-08 21:01:41,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-08 21:01:41,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-08 21:01:41,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-08 21:01:41,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-08 21:01:41,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-08 21:01:41,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-08 21:01:41,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-08 21:01:41,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-08 21:01:41,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-08 21:01:41,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-08 21:01:41,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-08 21:01:41,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-08 21:01:41,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-08 21:01:41,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-08 21:01:41,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-08 21:01:41,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-08 21:01:41,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-08 21:01:41,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-08 21:01:41,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-08 21:01:41,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-08 21:01:41,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-08 21:01:41,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-08 21:01:41,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-08 21:01:41,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-08 21:01:41,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-08 21:01:41,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-08 21:01:41,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-08 21:01:41,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-08 21:01:41,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-08 21:01:41,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-08 21:01:41,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-08 21:01:41,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-08 21:01:41,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-08 21:01:41,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-08 21:01:41,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-08 21:01:41,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-08 21:01:41,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-08 21:01:41,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-08 21:01:41,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-08 21:01:41,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-08 21:01:41,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-08 21:01:41,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-08 21:01:41,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-08 21:01:41,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-08 21:01:41,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-08 21:01:41,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-08 21:01:41,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-08 21:01:41,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-08 21:01:41,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-08 21:01:41,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-08 21:01:41,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-08 21:01:41,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-08 21:01:41,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-08 21:01:41,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-08 21:01:41,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-08 21:01:41,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-08 21:01:41,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-08 21:01:41,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-08 21:01:41,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-08 21:01:41,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-08 21:01:41,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-08 21:01:41,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-08 21:01:41,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-08 21:01:41,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-08 21:01:41,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-08 21:01:41,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-08 21:01:41,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-08 21:01:41,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-08 21:01:41,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-08 21:01:41,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-08 21:01:41,194 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022049270859364138); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03399016164516465, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022049270859364138); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 21:01:41,195 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-08 21:01:41,195 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013890887469044074, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022049270859364138); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03399016164516465, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022049270859364138); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-08 21:01:41,351 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 240 ms to try 5000 different iterations.  Found a solution that moves 994 regions; Going from a computed imbalance of 0.013890887469044074 to a new imbalance of 0.0011196835582222722. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005877325962449793); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010907220191267141); MoveCostFunction : (multiplier=7.0, imbalance=0.11044444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005877325962449793); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 21:01:41,353 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv392842158:81 , srv1312946646:82 , srv601459500:82 , srv1171505731:83 , srv1176324266:83 , srv1844984463:83 , srv471392759:83 , srv1142695142:84 , srv1270699725:84 , srv1389655910:84 , srv139556414:84 , srv1439648029:84 , srv1558363389:84 , srv1606346206:84 , srv1623022434:84 , srv1757502019:84 , srv1762696074:84 , srv1940322592:84 , srv2078603354:84 , srv354622703:84 , srv522983751:84 , srv589151788:84 , srv703581742:84 , srv882947513:84 , srv993207587:84 , srv106482882:85 , srv1127694968:85 , srv1340852699:85 , srv1378038673:85 , srv1470917578:85 , srv1521769290:85 , srv1643591997:85 , srv1730610746:85 , srv1744571159:85 , srv1768560338:85 , srv1975344943:85 , srv1995523839:85 , srv2017989594:85 , srv2092490940:85 , srv2108467319:85 , srv311089809:85 , srv35683316:85 , srv401534877:85 , srv479625257:85 , srv500943747:85 , srv564833147:85 , srv565986631:85 , srv599491235:85 , srv62241484:85 , srv762121503:85 , srv924199915:85 , srv1197400377:86 , srv1199002850:86 , srv1218014158:86 , srv1322078838:86 , srv1399479203:86 , srv1589728830:86 , srv1753064799:86 , srv1916375045:86 , srv1993378048:86 , srv2036065400:86 , srv2142520698:86 , srv219562644:86 , srv268008008:86 , srv270580743:86 , srv273491036:86 , srv35288526:86 , srv46669828:86 , srv529655867:86 , srv536633585:86 , srv702614480:86 , srv745188519:86 , srv775834625:86 , srv780059868:86 , srv819111366:86 , srv974382741:86 , srv98145949:86 , srv1221466471:87 , srv122978775:87 , srv1247520683:87 , srv1392296251:87 , srv1408086951:87 , srv1556745260:87 , srv1767083400:87 , srv1969739994:87 , srv431057273:87 , srv499539334:87 , srv571732148:87 , srv571858677:87 , srv584649307:87 , srv647135200:87 , srv1959710238:88 , srv1979171709:88 , srv2131413470:88 , srv426494307:88 , srv56476101:88 , srv703660695:88 , srv1039636446:89 , srv1418536510:90 , srv1928295909:540 }
2023-04-08 21:01:41,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-08 21:01:41,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv702614480=87, srv924199915=96, srv1408086951=21, srv1928295909=41, srv745188519=90, srv2131413470=54, srv1197400377=6, srv1399479203=20, srv1993378048=47, srv1142695142=3, srv270580743=58, srv529655867=74, srv565986631=78, srv1940322592=42, srv1606346206=29, srv35288526=61, srv1270699725=12, srv1556745260=26, srv392842158=64, srv139556414=19, srv780059868=93, srv479625257=70, srv2108467319=53, srv431057273=67, srv1470917578=24, srv1340852699=15, srv703660695=89, srv1521769290=25, srv268008008=57, srv647135200=86, srv1039636446=0, srv1199002850=7, srv1218014158=8, srv1221466471=9, srv564833147=77, srv1389655910=17, srv882947513=95, srv2078603354=51, srv2017989594=49, srv122978775=10, srv1171505731=4, srv401534877=65, srv1392296251=18, srv1247520683=11, srv1744571159=33, srv1127694968=2, srv1623022434=30, srv536633585=75, srv599491235=83, srv1589728830=28, srv601459500=84, srv2092490940=52, srv2036065400=50, srv1322078838=14, srv775834625=92, srv571732148=79, srv1762696074=36, srv219562644=56, srv1979171709=46, srv993207587=99, srv471392759=69, srv1643591997=31, srv1558363389=27, srv98145949=98, srv106482882=1, srv1439648029=23, srv273491036=59, srv974382741=97, srv584649307=81, srv1767083400=37, srv762121503=91, srv354622703=62, srv426494307=66, srv1176324266=5, srv311089809=60, srv1312946646=13, srv1969739994=44, srv703581742=88, srv56476101=76, srv1975344943=45, srv1418536510=22, srv46669828=68, srv1844984463=39, srv571858677=80, srv1730610746=32, srv1757502019=35, srv2142520698=55, srv500943747=72, srv1995523839=48, srv589151788=82, srv1768560338=38, srv62241484=85, srv522983751=73, srv819111366=94, srv499539334=71, srv35683316=63, srv1378038673=16, srv1959710238=43, srv1916375045=40, srv1753064799=34} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-08 21:01:41,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-08 21:01:41,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-08 21:01:41,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-08 21:01:41,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-08 21:01:41,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-08 21:01:41,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-08 21:01:41,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-08 21:01:41,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-08 21:01:41,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-08 21:01:41,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-08 21:01:41,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-08 21:01:41,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-08 21:01:41,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-08 21:01:41,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-08 21:01:41,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-08 21:01:41,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-08 21:01:41,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-08 21:01:41,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-08 21:01:41,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-08 21:01:41,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-08 21:01:41,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-08 21:01:41,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-08 21:01:41,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-08 21:01:41,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-08 21:01:41,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-08 21:01:41,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-08 21:01:41,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-08 21:01:41,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-08 21:01:41,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-08 21:01:41,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-08 21:01:41,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-08 21:01:41,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-08 21:01:41,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-08 21:01:41,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-08 21:01:41,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-08 21:01:41,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-08 21:01:41,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-08 21:01:41,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-08 21:01:41,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-08 21:01:41,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-08 21:01:41,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-08 21:01:41,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-08 21:01:41,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-08 21:01:41,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-08 21:01:41,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-08 21:01:41,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-08 21:01:41,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-08 21:01:41,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-08 21:01:41,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-08 21:01:41,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-08 21:01:41,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-08 21:01:41,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-08 21:01:41,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-08 21:01:41,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-08 21:01:41,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-08 21:01:41,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-08 21:01:41,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-08 21:01:41,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-08 21:01:41,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-08 21:01:41,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-08 21:01:41,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-08 21:01:41,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-08 21:01:41,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-08 21:01:41,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-08 21:01:41,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-08 21:01:41,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-08 21:01:41,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-08 21:01:41,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-08 21:01:41,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-08 21:01:41,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-08 21:01:41,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-08 21:01:41,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-08 21:01:41,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-08 21:01:41,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-08 21:01:41,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-08 21:01:41,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-08 21:01:41,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-08 21:01:41,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-08 21:01:41,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-08 21:01:41,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-08 21:01:41,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-08 21:01:41,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-08 21:01:41,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-08 21:01:41,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-08 21:01:41,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-08 21:01:41,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-08 21:01:41,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-08 21:01:41,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-08 21:01:41,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-08 21:01:41,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-08 21:01:41,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-08 21:01:41,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-08 21:01:41,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-08 21:01:41,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-08 21:01:41,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-08 21:01:41,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-08 21:01:41,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-08 21:01:41,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-08 21:01:41,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-08 21:01:41,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-08 21:01:41,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-08 21:01:41,463 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005877325962449793); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010907220191267141); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005877325962449793); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 21:01:41,463 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-08 21:01:41,464 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0011127234214602863, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005877325962449793); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010907220191267141); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005877325962449793); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-08 21:01:41,661 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 277 ms to try 5000 different iterations.  Found a solution that moves 323 regions; Going from a computed imbalance of 0.0011127234214602863 to a new imbalance of 2.316986744315495E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023688968483956715); MoveCostFunction : (multiplier=7.0, imbalance=0.03588888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 21:01:41,661 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1197400377:83 , srv1408086951:83 , srv1418536510:83 , srv2108467319:83 , srv571858677:83 , srv703660695:83 , srv1039636446:84 , srv106482882:84 , srv1127694968:84 , srv1171505731:84 , srv1176324266:84 , srv1218014158:84 , srv1221466471:84 , srv1399479203:84 , srv1439648029:84 , srv1470917578:84 , srv1521769290:84 , srv1556745260:84 , srv1643591997:84 , srv1767083400:84 , srv1916375045:84 , srv1969739994:84 , srv1975344943:84 , srv1979171709:84 , srv1993378048:84 , srv2017989594:84 , srv2092490940:84 , srv2131413470:84 , srv2142520698:84 , srv219562644:84 , srv268008008:84 , srv270580743:84 , srv273491036:84 , srv311089809:84 , srv35288526:84 , srv401534877:84 , srv46669828:84 , srv479625257:84 , srv56476101:84 , srv571732148:84 , srv584649307:84 , srv702614480:84 , srv745188519:84 , srv762121503:84 , srv775834625:84 , srv780059868:84 , srv924199915:84 , srv98145949:84 , srv1142695142:85 , srv1199002850:85 , srv122978775:85 , srv1247520683:85 , srv1270699725:85 , srv1312946646:85 , srv1322078838:85 , srv1340852699:85 , srv1378038673:85 , srv1389655910:85 , srv1392296251:85 , srv139556414:85 , srv1558363389:85 , srv1606346206:85 , srv1623022434:85 , srv1730610746:85 , srv1744571159:85 , srv1757502019:85 , srv1762696074:85 , srv1768560338:85 , srv1844984463:85 , srv1940322592:85 , srv1959710238:85 , srv1995523839:85 , srv2036065400:85 , srv2078603354:85 , srv35683316:85 , srv392842158:85 , srv426494307:85 , srv431057273:85 , srv499539334:85 , srv500943747:85 , srv522983751:85 , srv529655867:85 , srv536633585:85 , srv564833147:85 , srv565986631:85 , srv589151788:85 , srv599491235:85 , srv62241484:85 , srv647135200:85 , srv819111366:85 , srv882947513:85 , srv974382741:85 , srv993207587:85 , srv1589728830:86 , srv354622703:86 , srv471392759:86 , srv601459500:86 , srv703581742:86 , srv1753064799:87 , srv1928295909:632 }
2023-04-08 21:01:41,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-08 21:01:41,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv702614480=87, srv924199915=96, srv1408086951=21, srv1928295909=41, srv745188519=90, srv2131413470=54, srv1197400377=6, srv1399479203=20, srv1993378048=47, srv1142695142=3, srv270580743=58, srv529655867=74, srv565986631=78, srv1940322592=42, srv1606346206=29, srv35288526=61, srv1270699725=12, srv1556745260=26, srv392842158=64, srv139556414=19, srv780059868=93, srv479625257=70, srv2108467319=53, srv431057273=67, srv1470917578=24, srv1340852699=15, srv703660695=89, srv1521769290=25, srv268008008=57, srv647135200=86, srv1039636446=0, srv1199002850=7, srv1218014158=8, srv1221466471=9, srv564833147=77, srv1389655910=17, srv882947513=95, srv2078603354=51, srv2017989594=49, srv122978775=10, srv1171505731=4, srv401534877=65, srv1392296251=18, srv1247520683=11, srv1744571159=33, srv1127694968=2, srv1623022434=30, srv536633585=75, srv599491235=83, srv1589728830=28, srv601459500=84, srv2092490940=52, srv2036065400=50, srv1322078838=14, srv775834625=92, srv571732148=79, srv1762696074=36, srv219562644=56, srv1979171709=46, srv993207587=99, srv471392759=69, srv1643591997=31, srv1558363389=27, srv98145949=98, srv106482882=1, srv1439648029=23, srv273491036=59, srv974382741=97, srv584649307=81, srv1767083400=37, srv762121503=91, srv354622703=62, srv426494307=66, srv1176324266=5, srv311089809=60, srv1312946646=13, srv1969739994=44, srv703581742=88, srv56476101=76, srv1975344943=45, srv1418536510=22, srv46669828=68, srv1844984463=39, srv571858677=80, srv1730610746=32, srv1757502019=35, srv2142520698=55, srv500943747=72, srv1995523839=48, srv589151788=82, srv1768560338=38, srv62241484=85, srv522983751=73, srv819111366=94, srv499539334=71, srv35683316=63, srv1378038673=16, srv1959710238=43, srv1916375045=40, srv1753064799=34} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-08 21:01:41,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-08 21:01:41,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-08 21:01:41,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-08 21:01:41,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-08 21:01:41,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-08 21:01:41,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-08 21:01:41,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-08 21:01:41,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-08 21:01:41,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-08 21:01:41,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-08 21:01:41,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-08 21:01:41,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-08 21:01:41,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-08 21:01:41,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-08 21:01:41,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-08 21:01:41,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-08 21:01:41,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-08 21:01:41,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-08 21:01:41,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-08 21:01:41,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-08 21:01:41,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-08 21:01:41,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-08 21:01:41,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-08 21:01:41,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-08 21:01:41,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-08 21:01:41,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-08 21:01:41,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-08 21:01:41,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-08 21:01:41,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-08 21:01:41,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-08 21:01:41,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-08 21:01:41,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-08 21:01:41,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-08 21:01:41,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-08 21:01:41,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-08 21:01:41,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-08 21:01:41,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-08 21:01:41,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-08 21:01:41,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-08 21:01:41,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-08 21:01:41,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-08 21:01:41,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-08 21:01:41,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-08 21:01:41,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-08 21:01:41,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-08 21:01:41,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-08 21:01:41,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-08 21:01:41,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-08 21:01:41,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-08 21:01:41,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-08 21:01:41,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-08 21:01:41,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-08 21:01:41,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-08 21:01:41,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-08 21:01:41,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-08 21:01:41,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-08 21:01:41,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-08 21:01:41,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-08 21:01:41,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-08 21:01:41,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-08 21:01:41,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-08 21:01:41,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-08 21:01:41,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-08 21:01:41,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-08 21:01:41,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-08 21:01:41,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-08 21:01:41,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-08 21:01:41,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-08 21:01:41,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-08 21:01:41,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-08 21:01:41,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-08 21:01:41,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-08 21:01:41,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-08 21:01:41,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-08 21:01:41,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-08 21:01:41,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-08 21:01:41,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-08 21:01:41,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-08 21:01:41,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-08 21:01:41,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-08 21:01:41,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-08 21:01:41,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-08 21:01:41,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-08 21:01:41,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-08 21:01:41,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-08 21:01:41,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-08 21:01:41,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-08 21:01:41,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-08 21:01:41,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-08 21:01:41,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-08 21:01:41,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-08 21:01:41,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-08 21:01:41,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-08 21:01:41,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-08 21:01:41,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-08 21:01:41,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-08 21:01:41,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-08 21:01:41,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-08 21:01:41,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-08 21:01:41,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-08 21:01:41,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-08 21:01:41,783 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=2.090817310299252E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023688968483956715); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 21:01:41,783 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1197400377:83 , srv1408086951:83 , srv1418536510:83 , srv2108467319:83 , srv571858677:83 , srv703660695:83 , srv1039636446:84 , srv106482882:84 , srv1127694968:84 , srv1171505731:84 , srv1176324266:84 , srv1218014158:84 , srv1221466471:84 , srv1399479203:84 , srv1439648029:84 , srv1470917578:84 , srv1521769290:84 , srv1556745260:84 , srv1643591997:84 , srv1767083400:84 , srv1916375045:84 , srv1969739994:84 , srv1975344943:84 , srv1979171709:84 , srv1993378048:84 , srv2017989594:84 , srv2092490940:84 , srv2131413470:84 , srv2142520698:84 , srv219562644:84 , srv268008008:84 , srv270580743:84 , srv273491036:84 , srv311089809:84 , srv35288526:84 , srv401534877:84 , srv46669828:84 , srv479625257:84 , srv56476101:84 , srv571732148:84 , srv584649307:84 , srv702614480:84 , srv745188519:84 , srv762121503:84 , srv775834625:84 , srv780059868:84 , srv924199915:84 , srv98145949:84 , srv1142695142:85 , srv1199002850:85 , srv122978775:85 , srv1247520683:85 , srv1270699725:85 , srv1312946646:85 , srv1322078838:85 , srv1340852699:85 , srv1378038673:85 , srv1389655910:85 , srv1392296251:85 , srv139556414:85 , srv1558363389:85 , srv1606346206:85 , srv1623022434:85 , srv1730610746:85 , srv1744571159:85 , srv1757502019:85 , srv1762696074:85 , srv1768560338:85 , srv1844984463:85 , srv1940322592:85 , srv1959710238:85 , srv1995523839:85 , srv2036065400:85 , srv2078603354:85 , srv35683316:85 , srv392842158:85 , srv426494307:85 , srv431057273:85 , srv499539334:85 , srv500943747:85 , srv522983751:85 , srv529655867:85 , srv536633585:85 , srv564833147:85 , srv565986631:85 , srv589151788:85 , srv599491235:85 , srv62241484:85 , srv647135200:85 , srv819111366:85 , srv882947513:85 , srv974382741:85 , srv993207587:85 , srv1589728830:86 , srv354622703:86 , srv471392759:86 , srv601459500:86 , srv703581742:86 , srv1753064799:87 , srv1928295909:632 }
2023-04-08 21:01:41,812 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=159 (was 159), ProcessCount=167 (was 165) - ProcessCount LEAK? -, AvailableMemoryMB=4487 (was 4922)
2023-04-08 21:01:41,818 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=159, ProcessCount=167, AvailableMemoryMB=4487
2023-04-08 21:01:41,819 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-08 21:01:41,820 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-04-08 21:01:41,833 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1032877282:84 , srv1234030834:84 , srv1245233560:84 , srv1276889147:84 , srv1355499107:84 , srv1365166690:84 , srv1545030575:84 , srv1549946140:84 , srv1560486520:84 , srv1718144003:84 , srv1813545888:84 , srv1915970918:84 , srv1935202155:84 , srv2090960209:84 , srv2091507840:84 , srv2137106803:84 , srv231600246:84 , srv260485658:84 , srv260818788:84 , srv264530651:84 , srv398825796:84 , srv451850120:84 , srv553339641:84 , srv649147571:84 , srv71516118:84 , srv715179452:84 , srv748495513:84 , srv793791120:84 , srv803530939:84 , srv450615618:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-04-08 21:01:41,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-08 21:01:41,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1032877282=0, srv1915970918=11, srv1813545888=10, srv2090960209=13, srv1355499107=4, srv1718144003=9, srv1545030575=6, srv803530939=29, srv451850120=22, srv715179452=26, srv1245233560=2, srv1234030834=1, srv450615618=21, srv231600246=16, srv1935202155=12, srv553339641=23, srv1549946140=7, srv71516118=25, srv1560486520=8, srv1276889147=3, srv260485658=17, srv260818788=18, srv748495513=27, srv264530651=19, srv793791120=28, srv2091507840=14, srv1365166690=5, srv649147571=24, srv398825796=20, srv2137106803=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-08 21:01:41,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-08 21:01:41,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-08 21:01:41,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-08 21:01:41,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-08 21:01:41,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-08 21:01:41,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-08 21:01:41,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-08 21:01:41,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-08 21:01:41,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-08 21:01:41,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-08 21:01:41,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-08 21:01:41,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-08 21:01:41,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-08 21:01:41,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-08 21:01:41,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-08 21:01:41,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-08 21:01:41,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-08 21:01:41,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-08 21:01:41,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-08 21:01:41,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-08 21:01:41,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-08 21:01:41,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-08 21:01:41,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-08 21:01:41,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-08 21:01:41,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-08 21:01:41,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-08 21:01:41,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-08 21:01:41,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-08 21:01:41,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-08 21:01:41,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-08 21:01:41,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-08 21:01:41,878 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.099873104582469, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 21:01:41,878 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912507740155662, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.099873104582469, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-04-08 21:03:41,856 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9361372 different iterations.  Found a solution that moves 1843 regions; Going from a computed imbalance of 0.9912507740155662 to a new imbalance of 4.3016539410932495E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6825925925925926, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 21:03:41,863 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1032877282:90 , srv1234030834:90 , srv1245233560:90 , srv1276889147:90 , srv1355499107:90 , srv1365166690:90 , srv1545030575:90 , srv1549946140:90 , srv1560486520:90 , srv1718144003:90 , srv1813545888:90 , srv1915970918:90 , srv1935202155:90 , srv2090960209:90 , srv2091507840:90 , srv2137106803:90 , srv231600246:90 , srv260485658:90 , srv260818788:90 , srv264530651:90 , srv398825796:90 , srv450615618:90 , srv451850120:90 , srv553339641:90 , srv649147571:90 , srv71516118:90 , srv715179452:90 , srv748495513:90 , srv793791120:90 , srv803530939:90 }
2023-04-08 21:03:41,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-08 21:03:41,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1032877282=0, srv1915970918=11, srv1813545888=10, srv2090960209=13, srv1355499107=4, srv1718144003=9, srv1545030575=6, srv803530939=29, srv451850120=22, srv715179452=26, srv1245233560=2, srv1234030834=1, srv450615618=21, srv231600246=16, srv1935202155=12, srv553339641=23, srv1549946140=7, srv71516118=25, srv1560486520=8, srv1276889147=3, srv260485658=17, srv260818788=18, srv748495513=27, srv264530651=19, srv793791120=28, srv2091507840=14, srv1365166690=5, srv649147571=24, srv398825796=20, srv2137106803=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-08 21:03:41,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-08 21:03:41,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-08 21:03:41,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-08 21:03:41,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-08 21:03:41,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-08 21:03:41,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-08 21:03:41,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-08 21:03:41,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-08 21:03:41,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-08 21:03:41,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-08 21:03:41,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-08 21:03:41,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-08 21:03:41,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-08 21:03:41,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-08 21:03:41,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-08 21:03:41,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-08 21:03:41,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-08 21:03:41,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-08 21:03:41,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-08 21:03:41,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-08 21:03:41,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-08 21:03:41,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-08 21:03:41,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-08 21:03:41,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-08 21:03:41,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-08 21:03:41,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-08 21:03:41,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-08 21:03:41,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-08 21:03:41,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-08 21:03:41,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-08 21:03:41,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-08 21:03:41,882 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 21:03:41,882 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1032877282:90 , srv1234030834:90 , srv1245233560:90 , srv1276889147:90 , srv1355499107:90 , srv1365166690:90 , srv1545030575:90 , srv1549946140:90 , srv1560486520:90 , srv1718144003:90 , srv1813545888:90 , srv1915970918:90 , srv1935202155:90 , srv2090960209:90 , srv2091507840:90 , srv2137106803:90 , srv231600246:90 , srv260485658:90 , srv260818788:90 , srv264530651:90 , srv398825796:90 , srv450615618:90 , srv451850120:90 , srv553339641:90 , srv649147571:90 , srv71516118:90 , srv715179452:90 , srv748495513:90 , srv793791120:90 , srv803530939:90 }
2023-04-08 21:03:41,892 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=269 (was 159) - SystemLoadAverage LEAK? -, ProcessCount=165 (was 167), AvailableMemoryMB=4502 (was 4487) - AvailableMemoryMB LEAK? -