2023-05-11 08:53:31,751 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-05-11 08:53:32,084 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-05-11 08:53:32,208 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-05-11 08:53:32,269 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-11 08:53:32,276 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-05-11 08:53:32,291 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=308, ProcessCount=175, AvailableMemoryMB=9862
2023-05-11 08:53:32,293 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-11 08:53:32,293 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-05-11 08:53:32,431 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1031328482:84 , srv105233554:84 , srv106571720:84 , srv1066857067:84 , srv1076729323:84 , srv1112599708:84 , srv1157265950:84 , srv1174895264:84 , srv1200484306:84 , srv1213954116:84 , srv1222385931:84 , srv1300121444:84 , srv1340409870:84 , srv1370321424:84 , srv1396926753:84 , srv1416718888:84 , srv1422187148:84 , srv1430903384:84 , srv1442255124:84 , srv1442495696:84 , srv1450205080:84 , srv1502686865:84 , srv152653220:84 , srv1527410656:84 , srv1544057090:84 , srv1561560914:84 , srv1584987150:84 , srv1642079494:84 , srv1676708918:84 , srv1707961612:84 , srv1731346562:84 , srv1751610126:84 , srv1770204534:84 , srv177588469:84 , srv1805196964:84 , srv1808342035:84 , srv1835241327:84 , srv1880098460:84 , srv1889002841:84 , srv1891142648:84 , srv1946430809:84 , srv1958473037:84 , srv1980755879:84 , srv1986875100:84 , srv1992914079:84 , srv201487997:84 , srv2017131521:84 , srv2072218809:84 , srv2075176166:84 , srv2087195590:84 , srv2091765922:84 , srv2121361670:84 , srv2134611214:84 , srv228330115:84 , srv248783672:84 , srv251885611:84 , srv279192051:84 , srv324620804:84 , srv325393181:84 , srv331825714:84 , srv343249675:84 , srv348360542:84 , srv358852285:84 , srv382522915:84 , srv389185882:84 , srv402044541:84 , srv413326508:84 , srv47424183:84 , srv47491821:84 , srv49365122:84 , srv495317827:84 , srv496268501:84 , srv514631293:84 , srv526844331:84 , srv530448304:84 , srv539454120:84 , srv561308370:84 , srv577385338:84 , srv588440441:84 , srv612493538:84 , srv620862355:84 , srv632545587:84 , srv66832939:84 , srv704472988:84 , srv715736950:84 , srv756457572:84 , srv762266303:84 , srv763434465:84 , srv813275997:84 , srv816486490:84 , srv828048214:84 , srv83112443:84 , srv850142140:84 , srv870294503:84 , srv874933324:84 , srv903820536:84 , srv906052362:84 , srv910300102:84 , srv925281138:84 , srv751679753:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-05-11 08:53:32,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-11 08:53:32,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1980755879=42, srv2017131521=46, srv496268501=71, srv1066857067=3, srv228330115=53, srv715736950=84, srv539454120=75, srv1422187148=16, srv751679753=85, srv105233554=1, srv402044541=65, srv1544057090=24, srv756457572=86, srv1031328482=0, srv530448304=74, srv828048214=91, srv925281138=99, srv1416718888=15, srv2087195590=49, srv2134611214=52, srv47424183=67, srv1300121444=11, srv612493538=79, srv874933324=95, srv903820536=96, srv152653220=22, srv1112599708=5, srv331825714=59, srv1561560914=25, srv325393181=58, srv816486490=90, srv1676708918=28, srv1835241327=36, srv343249675=60, srv910300102=98, srv763434465=88, srv1174895264=7, srv1880098460=37, srv49365122=69, srv1889002841=38, srv201487997=45, srv251885611=55, srv1213954116=9, srv324620804=57, srv495317827=70, srv2091765922=50, srv248783672=54, srv906052362=97, srv1222385931=10, srv1396926753=14, srv561308370=76, srv1370321424=13, srv389185882=64, srv83112443=92, srv1527410656=23, srv348360542=61, srv358852285=62, srv1442255124=18, srv413326508=66, srv1808342035=35, srv66832939=82, srv279192051=56, srv704472988=83, srv1805196964=34, srv514631293=72, srv813275997=89, srv762266303=87, srv1430903384=17, srv1200484306=8, srv1502686865=21, srv526844331=73, srv47491821=68, srv1958473037=41, srv1076729323=4, srv1642079494=27, srv620862355=80, srv850142140=93, srv1891142648=39, srv577385338=77, srv177588469=33, srv632545587=81, srv870294503=94, srv106571720=2, srv1986875100=43, srv1442495696=19, srv1731346562=30, srv2072218809=47, srv1707961612=29, srv2121361670=51, srv2075176166=48, srv588440441=78, srv1770204534=32, srv1946430809=40, srv1450205080=20, srv1992914079=44, srv1751610126=31, srv382522915=63, srv1157265950=6, srv1340409870=12, srv1584987150=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-11 08:53:32,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-11 08:53:32,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-11 08:53:32,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-11 08:53:32,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-11 08:53:32,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-11 08:53:32,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-11 08:53:32,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-11 08:53:32,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-11 08:53:32,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-11 08:53:32,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-11 08:53:32,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-11 08:53:32,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-11 08:53:32,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-11 08:53:32,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-11 08:53:32,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-11 08:53:32,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-11 08:53:32,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-11 08:53:32,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-11 08:53:32,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-11 08:53:32,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-11 08:53:32,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-11 08:53:32,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-11 08:53:32,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-11 08:53:32,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-11 08:53:32,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-11 08:53:32,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-11 08:53:32,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-11 08:53:32,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-11 08:53:32,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-11 08:53:32,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-11 08:53:32,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-11 08:53:32,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-11 08:53:32,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-11 08:53:32,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-11 08:53:32,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-11 08:53:32,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-11 08:53:32,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-11 08:53:32,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-11 08:53:32,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-11 08:53:32,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-11 08:53:32,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-11 08:53:32,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-11 08:53:32,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-11 08:53:32,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-11 08:53:32,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-11 08:53:32,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-11 08:53:32,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-11 08:53:32,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-11 08:53:32,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-11 08:53:32,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-11 08:53:32,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-11 08:53:32,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-11 08:53:32,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-11 08:53:32,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-11 08:53:32,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-11 08:53:32,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-11 08:53:32,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-11 08:53:32,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-11 08:53:32,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-11 08:53:32,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-11 08:53:32,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-11 08:53:32,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-11 08:53:32,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-11 08:53:32,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-11 08:53:32,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-11 08:53:32,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-11 08:53:32,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-11 08:53:32,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-11 08:53:32,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-11 08:53:32,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-11 08:53:32,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-11 08:53:32,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-11 08:53:32,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-11 08:53:32,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-11 08:53:32,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-11 08:53:32,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-11 08:53:32,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-11 08:53:32,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-11 08:53:32,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-11 08:53:32,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-11 08:53:32,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-11 08:53:32,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-11 08:53:32,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-11 08:53:32,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-11 08:53:32,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-11 08:53:32,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-11 08:53:32,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-11 08:53:32,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-11 08:53:32,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-11 08:53:32,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-11 08:53:32,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-11 08:53:32,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-11 08:53:32,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-11 08:53:32,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-11 08:53:32,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-11 08:53:32,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-11 08:53:32,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-11 08:53:32,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-11 08:53:32,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-11 08:53:32,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-11 08:53:32,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-11 08:53:32,802 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-11 08:53:32,803 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-11 08:53:32,803 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-11 08:53:33,195 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 610 ms to try 5000 different iterations.  Found a solution that moves 4682 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17811210458475754. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04220686116696286, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05920536269580348, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5202222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04220686116696286, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17308333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.242, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-11 08:53:33,226 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv152653220:81 , srv47424183:81 , srv496268501:81 , srv1808342035:82 , srv1835241327:82 , srv1891142648:82 , srv47491821:82 , srv1340409870:83 , srv1370321424:83 , srv177588469:83 , srv2134611214:83 , srv382522915:83 , srv526844331:83 , srv612493538:83 , srv762266303:83 , srv1300121444:84 , srv1416718888:84 , srv2017131521:84 , srv2072218809:84 , srv248783672:84 , srv279192051:84 , srv324620804:84 , srv331825714:84 , srv539454120:84 , srv561308370:84 , srv66832939:84 , srv903820536:84 , srv910300102:84 , srv1066857067:85 , srv1174895264:85 , srv1200484306:85 , srv1422187148:85 , srv1707961612:85 , srv1731346562:85 , srv1770204534:85 , srv1880098460:85 , srv1946430809:85 , srv2075176166:85 , srv343249675:85 , srv389185882:85 , srv495317827:85 , srv632545587:85 , srv763434465:85 , srv816486490:85 , srv1031328482:86 , srv1076729323:86 , srv1396926753:86 , srv1502686865:86 , srv1544057090:86 , srv1584987150:86 , srv1676708918:86 , srv1889002841:86 , srv1980755879:86 , srv201487997:86 , srv358852285:86 , srv577385338:86 , srv715736950:86 , srv813275997:86 , srv906052362:86 , srv925281138:86 , srv106571720:87 , srv1527410656:87 , srv1642079494:87 , srv1958473037:87 , srv2091765922:87 , srv514631293:87 , srv620862355:87 , srv756457572:87 , srv1430903384:88 , srv1442255124:88 , srv1450205080:88 , srv1751610126:88 , srv1992914079:88 , srv2087195590:88 , srv2121361670:88 , srv251885611:88 , srv325393181:88 , srv402044541:88 , srv413326508:88 , srv530448304:88 , srv828048214:88 , srv1222385931:89 , srv1986875100:89 , srv228330115:89 , srv348360542:89 , srv49365122:89 , srv83112443:89 , srv870294503:89 , srv105233554:90 , srv1442495696:90 , srv1805196964:90 , srv704472988:90 , srv874933324:90 , srv1157265950:91 , srv588440441:91 , srv1561560914:93 , srv850142140:93 , srv1112599708:95 , srv1213954116:96 , srv751679753:465 }
2023-05-11 08:53:33,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-11 08:53:33,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1980755879=42, srv2017131521=46, srv496268501=71, srv1066857067=3, srv228330115=53, srv715736950=84, srv539454120=75, srv1422187148=16, srv751679753=85, srv105233554=1, srv402044541=65, srv1544057090=24, srv756457572=86, srv1031328482=0, srv530448304=74, srv828048214=91, srv925281138=99, srv1416718888=15, srv2087195590=49, srv2134611214=52, srv47424183=67, srv1300121444=11, srv612493538=79, srv874933324=95, srv903820536=96, srv152653220=22, srv1112599708=5, srv331825714=59, srv1561560914=25, srv325393181=58, srv816486490=90, srv1676708918=28, srv1835241327=36, srv343249675=60, srv910300102=98, srv763434465=88, srv1174895264=7, srv1880098460=37, srv49365122=69, srv1889002841=38, srv201487997=45, srv251885611=55, srv1213954116=9, srv324620804=57, srv495317827=70, srv2091765922=50, srv248783672=54, srv906052362=97, srv1222385931=10, srv1396926753=14, srv561308370=76, srv1370321424=13, srv389185882=64, srv83112443=92, srv1527410656=23, srv348360542=61, srv358852285=62, srv1442255124=18, srv413326508=66, srv1808342035=35, srv66832939=82, srv279192051=56, srv704472988=83, srv1805196964=34, srv514631293=72, srv813275997=89, srv762266303=87, srv1430903384=17, srv1200484306=8, srv1502686865=21, srv526844331=73, srv47491821=68, srv1958473037=41, srv1076729323=4, srv1642079494=27, srv620862355=80, srv850142140=93, srv1891142648=39, srv577385338=77, srv177588469=33, srv632545587=81, srv870294503=94, srv106571720=2, srv1986875100=43, srv1442495696=19, srv1731346562=30, srv2072218809=47, srv1707961612=29, srv2121361670=51, srv2075176166=48, srv588440441=78, srv1770204534=32, srv1946430809=40, srv1450205080=20, srv1992914079=44, srv1751610126=31, srv382522915=63, srv1157265950=6, srv1340409870=12, srv1584987150=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-11 08:53:33,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-11 08:53:33,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-11 08:53:33,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-11 08:53:33,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-11 08:53:33,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-11 08:53:33,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-11 08:53:33,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-11 08:53:33,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-11 08:53:33,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-11 08:53:33,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-11 08:53:33,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-11 08:53:33,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-11 08:53:33,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-11 08:53:33,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-11 08:53:33,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-11 08:53:33,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-11 08:53:33,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-11 08:53:33,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-11 08:53:33,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-11 08:53:33,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-11 08:53:33,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-11 08:53:33,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-11 08:53:33,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-11 08:53:33,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-11 08:53:33,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-11 08:53:33,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-11 08:53:33,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-11 08:53:33,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-11 08:53:33,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-11 08:53:33,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-11 08:53:33,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-11 08:53:33,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-11 08:53:33,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-11 08:53:33,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-11 08:53:33,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-11 08:53:33,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-11 08:53:33,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-11 08:53:33,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-11 08:53:33,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-11 08:53:33,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-11 08:53:33,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-11 08:53:33,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-11 08:53:33,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-11 08:53:33,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-11 08:53:33,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-11 08:53:33,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-11 08:53:33,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-11 08:53:33,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-11 08:53:33,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-11 08:53:33,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-11 08:53:33,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-11 08:53:33,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-11 08:53:33,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-11 08:53:33,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-11 08:53:33,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-11 08:53:33,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-11 08:53:33,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-11 08:53:33,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-11 08:53:33,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-11 08:53:33,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-11 08:53:33,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-11 08:53:33,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-11 08:53:33,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-11 08:53:33,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-11 08:53:33,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-11 08:53:33,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-11 08:53:33,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-11 08:53:33,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-11 08:53:33,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-11 08:53:33,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-11 08:53:33,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-11 08:53:33,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-11 08:53:33,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-11 08:53:33,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-11 08:53:33,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-11 08:53:33,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-11 08:53:33,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-11 08:53:33,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-11 08:53:33,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-11 08:53:33,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-11 08:53:33,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-11 08:53:33,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-11 08:53:33,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-11 08:53:33,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-11 08:53:33,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-11 08:53:33,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-11 08:53:33,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-11 08:53:33,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-11 08:53:33,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-11 08:53:33,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-11 08:53:33,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-11 08:53:33,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-11 08:53:33,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-11 08:53:33,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-11 08:53:33,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-11 08:53:33,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-11 08:53:33,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-11 08:53:33,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-11 08:53:33,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-11 08:53:33,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-11 08:53:33,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-11 08:53:33,387 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04220686116696286, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05920536269580348, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04220686116696286, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17308333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.242, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-11 08:53:33,387 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-11 08:53:33,388 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1780793205200497, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04220686116696286, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05920536269580348, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04220686116696286, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17308333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.242, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-11 08:53:33,721 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 441 ms to try 5000 different iterations.  Found a solution that moves 2695 regions; Going from a computed imbalance of 0.1780793205200497 to a new imbalance of 0.014720278064827286. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022166034465498707); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032258774113335026, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29944444444444446, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022166034465498707); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0455, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-11 08:53:33,732 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1213954116:76 , srv1112599708:80 , srv1561560914:80 , srv1805196964:80 , srv588440441:80 , srv874933324:80 , srv105233554:81 , srv1157265950:81 , srv1442495696:81 , srv1986875100:81 , srv49365122:81 , srv704472988:81 , srv850142140:81 , srv1222385931:82 , srv228330115:82 , srv348360542:82 , srv413326508:82 , srv1430903384:83 , srv1751610126:83 , srv2087195590:83 , srv402044541:83 , srv83112443:83 , srv870294503:83 , srv1031328482:84 , srv106571720:84 , srv1527410656:84 , srv1544057090:84 , srv1770204534:84 , srv2091765922:84 , srv251885611:84 , srv325393181:84 , srv530448304:84 , srv577385338:84 , srv1422187148:85 , srv1502686865:85 , srv201487997:85 , srv2121361670:85 , srv828048214:85 , srv1396926753:86 , srv1442255124:86 , srv1642079494:86 , srv1731346562:86 , srv1880098460:86 , srv1992914079:86 , srv358852285:86 , srv561308370:86 , srv66832939:86 , srv813275997:86 , srv925281138:86 , srv1450205080:87 , srv1584987150:87 , srv1980755879:87 , srv2075176166:87 , srv331825714:87 , srv389185882:87 , srv514631293:87 , srv620862355:87 , srv632545587:87 , srv756457572:87 , srv762266303:87 , srv763434465:87 , srv816486490:87 , srv906052362:87 , srv1066857067:88 , srv1076729323:88 , srv1174895264:88 , srv1200484306:88 , srv1300121444:88 , srv1676708918:88 , srv1808342035:88 , srv1835241327:88 , srv1889002841:88 , srv1946430809:88 , srv1958473037:88 , srv2017131521:88 , srv2134611214:88 , srv279192051:88 , srv324620804:88 , srv343249675:88 , srv495317827:88 , srv526844331:88 , srv539454120:88 , srv910300102:88 , srv1340409870:89 , srv1416718888:89 , srv177588469:89 , srv1891142648:89 , srv382522915:89 , srv612493538:89 , srv715736950:89 , srv152653220:90 , srv1707961612:90 , srv2072218809:90 , srv248783672:90 , srv47491821:90 , srv903820536:90 , srv1370321424:91 , srv47424183:91 , srv496268501:91 , srv751679753:506 }
2023-05-11 08:53:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-11 08:53:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1980755879=42, srv2017131521=46, srv496268501=71, srv1066857067=3, srv228330115=53, srv715736950=84, srv539454120=75, srv1422187148=16, srv751679753=85, srv105233554=1, srv402044541=65, srv1544057090=24, srv756457572=86, srv1031328482=0, srv530448304=74, srv828048214=91, srv925281138=99, srv1416718888=15, srv2087195590=49, srv2134611214=52, srv47424183=67, srv1300121444=11, srv612493538=79, srv874933324=95, srv903820536=96, srv152653220=22, srv1112599708=5, srv331825714=59, srv1561560914=25, srv325393181=58, srv816486490=90, srv1676708918=28, srv1835241327=36, srv343249675=60, srv910300102=98, srv763434465=88, srv1174895264=7, srv1880098460=37, srv49365122=69, srv1889002841=38, srv201487997=45, srv251885611=55, srv1213954116=9, srv324620804=57, srv495317827=70, srv2091765922=50, srv248783672=54, srv906052362=97, srv1222385931=10, srv1396926753=14, srv561308370=76, srv1370321424=13, srv389185882=64, srv83112443=92, srv1527410656=23, srv348360542=61, srv358852285=62, srv1442255124=18, srv413326508=66, srv1808342035=35, srv66832939=82, srv279192051=56, srv704472988=83, srv1805196964=34, srv514631293=72, srv813275997=89, srv762266303=87, srv1430903384=17, srv1200484306=8, srv1502686865=21, srv526844331=73, srv47491821=68, srv1958473037=41, srv1076729323=4, srv1642079494=27, srv620862355=80, srv850142140=93, srv1891142648=39, srv577385338=77, srv177588469=33, srv632545587=81, srv870294503=94, srv106571720=2, srv1986875100=43, srv1442495696=19, srv1731346562=30, srv2072218809=47, srv1707961612=29, srv2121361670=51, srv2075176166=48, srv588440441=78, srv1770204534=32, srv1946430809=40, srv1450205080=20, srv1992914079=44, srv1751610126=31, srv382522915=63, srv1157265950=6, srv1340409870=12, srv1584987150=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-11 08:53:33,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-11 08:53:33,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-11 08:53:33,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-11 08:53:33,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-11 08:53:33,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-11 08:53:33,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-11 08:53:33,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-11 08:53:33,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-11 08:53:33,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-11 08:53:33,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-11 08:53:33,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-11 08:53:33,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-11 08:53:33,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-11 08:53:33,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-11 08:53:33,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-11 08:53:33,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-11 08:53:33,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-11 08:53:33,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-11 08:53:33,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-11 08:53:33,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-11 08:53:33,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-11 08:53:33,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-11 08:53:33,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-11 08:53:33,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-11 08:53:33,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-11 08:53:33,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-11 08:53:33,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-11 08:53:33,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-11 08:53:33,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-11 08:53:33,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-11 08:53:33,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-11 08:53:33,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-11 08:53:33,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-11 08:53:33,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-11 08:53:33,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-11 08:53:33,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-11 08:53:33,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-11 08:53:33,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-11 08:53:33,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-11 08:53:33,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-11 08:53:33,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-11 08:53:33,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-11 08:53:33,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-11 08:53:33,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-11 08:53:33,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-11 08:53:33,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-11 08:53:33,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-11 08:53:33,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-11 08:53:33,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-11 08:53:33,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-11 08:53:33,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-11 08:53:33,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-11 08:53:33,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-11 08:53:33,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-11 08:53:33,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-11 08:53:33,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-11 08:53:33,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-11 08:53:33,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-11 08:53:33,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-11 08:53:33,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-11 08:53:33,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-11 08:53:33,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-11 08:53:33,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-11 08:53:33,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-11 08:53:33,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-11 08:53:33,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-11 08:53:33,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-11 08:53:33,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-11 08:53:33,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-11 08:53:33,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-11 08:53:33,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-11 08:53:33,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-11 08:53:33,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-11 08:53:33,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-11 08:53:33,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-11 08:53:33,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-11 08:53:33,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-11 08:53:33,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-11 08:53:33,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-11 08:53:33,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-11 08:53:33,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-11 08:53:33,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-11 08:53:33,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-11 08:53:33,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-11 08:53:33,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-11 08:53:33,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-11 08:53:33,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-11 08:53:33,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-11 08:53:33,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-11 08:53:33,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-11 08:53:33,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-11 08:53:33,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-11 08:53:33,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-11 08:53:33,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-11 08:53:33,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-11 08:53:33,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-11 08:53:33,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-11 08:53:33,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-11 08:53:33,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-11 08:53:33,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-11 08:53:33,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-11 08:53:33,932 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022166034465498707); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032258774113335026, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022166034465498707); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0455, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-11 08:53:33,933 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-11 08:53:33,933 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014701407271493732, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022166034465498707); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032258774113335026, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022166034465498707); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0455, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-11 08:53:34,295 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 508 ms to try 5000 different iterations.  Found a solution that moves 969 regions; Going from a computed imbalance of 0.014701407271493732 to a new imbalance of 0.0010585937299825072. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005984558423028655); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008926687311226545); MoveCostFunction : (multiplier=7.0, imbalance=0.10766666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005984558423028655); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=6.666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-11 08:53:34,297 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1112599708:82 , srv1707961612:82 , srv1958473037:82 , srv1076729323:83 , srv1450205080:83 , srv2072218809:83 , srv248783672:83 , srv514631293:83 , srv1200484306:84 , srv1370321424:84 , srv1442255124:84 , srv1561560914:84 , srv1676708918:84 , srv1946430809:84 , srv1992914079:84 , srv343249675:84 , srv620862355:84 , srv715736950:84 , srv756457572:84 , srv828048214:84 , srv850142140:84 , srv903820536:84 , srv906052362:84 , srv1066857067:85 , srv1174895264:85 , srv1300121444:85 , srv1340409870:85 , srv1396926753:85 , srv1416718888:85 , srv1584987150:85 , srv1642079494:85 , srv177588469:85 , srv2075176166:85 , srv2121361670:85 , srv251885611:85 , srv324620804:85 , srv358852285:85 , srv389185882:85 , srv47424183:85 , srv495317827:85 , srv496268501:85 , srv539454120:85 , srv612493538:85 , srv632545587:85 , srv763434465:85 , srv816486490:85 , srv870294503:85 , srv925281138:85 , srv106571720:86 , srv1157265950:86 , srv1213954116:86 , srv1222385931:86 , srv1430903384:86 , srv1442495696:86 , srv1502686865:86 , srv152653220:86 , srv1527410656:86 , srv1731346562:86 , srv1889002841:86 , srv1891142648:86 , srv1980755879:86 , srv2017131521:86 , srv2134611214:86 , srv228330115:86 , srv279192051:86 , srv325393181:86 , srv331825714:86 , srv382522915:86 , srv47491821:86 , srv526844331:86 , srv530448304:86 , srv588440441:86 , srv813275997:86 , srv83112443:86 , srv910300102:86 , srv1031328482:87 , srv105233554:87 , srv1422187148:87 , srv1544057090:87 , srv1751610126:87 , srv1805196964:87 , srv1808342035:87 , srv1880098460:87 , srv201487997:87 , srv2087195590:87 , srv2091765922:87 , srv348360542:87 , srv402044541:87 , srv413326508:87 , srv49365122:87 , srv577385338:87 , srv66832939:87 , srv704472988:87 , srv762266303:87 , srv874933324:87 , srv1770204534:88 , srv1835241327:88 , srv1986875100:88 , srv561308370:88 , srv751679753:540 }
2023-05-11 08:53:34,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-11 08:53:34,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1980755879=42, srv2017131521=46, srv496268501=71, srv1066857067=3, srv228330115=53, srv715736950=84, srv539454120=75, srv1422187148=16, srv751679753=85, srv105233554=1, srv402044541=65, srv1544057090=24, srv756457572=86, srv1031328482=0, srv530448304=74, srv828048214=91, srv925281138=99, srv1416718888=15, srv2087195590=49, srv2134611214=52, srv47424183=67, srv1300121444=11, srv612493538=79, srv874933324=95, srv903820536=96, srv152653220=22, srv1112599708=5, srv331825714=59, srv1561560914=25, srv325393181=58, srv816486490=90, srv1676708918=28, srv1835241327=36, srv343249675=60, srv910300102=98, srv763434465=88, srv1174895264=7, srv1880098460=37, srv49365122=69, srv1889002841=38, srv201487997=45, srv251885611=55, srv1213954116=9, srv324620804=57, srv495317827=70, srv2091765922=50, srv248783672=54, srv906052362=97, srv1222385931=10, srv1396926753=14, srv561308370=76, srv1370321424=13, srv389185882=64, srv83112443=92, srv1527410656=23, srv348360542=61, srv358852285=62, srv1442255124=18, srv413326508=66, srv1808342035=35, srv66832939=82, srv279192051=56, srv704472988=83, srv1805196964=34, srv514631293=72, srv813275997=89, srv762266303=87, srv1430903384=17, srv1200484306=8, srv1502686865=21, srv526844331=73, srv47491821=68, srv1958473037=41, srv1076729323=4, srv1642079494=27, srv620862355=80, srv850142140=93, srv1891142648=39, srv577385338=77, srv177588469=33, srv632545587=81, srv870294503=94, srv106571720=2, srv1986875100=43, srv1442495696=19, srv1731346562=30, srv2072218809=47, srv1707961612=29, srv2121361670=51, srv2075176166=48, srv588440441=78, srv1770204534=32, srv1946430809=40, srv1450205080=20, srv1992914079=44, srv1751610126=31, srv382522915=63, srv1157265950=6, srv1340409870=12, srv1584987150=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-11 08:53:34,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-11 08:53:34,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-11 08:53:34,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-11 08:53:34,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-11 08:53:34,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-11 08:53:34,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-11 08:53:34,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-11 08:53:34,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-11 08:53:34,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-11 08:53:34,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-11 08:53:34,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-11 08:53:34,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-11 08:53:34,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-11 08:53:34,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-11 08:53:34,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-11 08:53:34,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-11 08:53:34,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-11 08:53:34,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-11 08:53:34,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-11 08:53:34,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-11 08:53:34,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-11 08:53:34,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-11 08:53:34,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-11 08:53:34,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-11 08:53:34,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-11 08:53:34,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-11 08:53:34,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-11 08:53:34,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-11 08:53:34,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-11 08:53:34,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-11 08:53:34,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-11 08:53:34,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-11 08:53:34,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-11 08:53:34,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-11 08:53:34,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-11 08:53:34,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-11 08:53:34,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-11 08:53:34,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-11 08:53:34,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-11 08:53:34,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-11 08:53:34,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-11 08:53:34,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-11 08:53:34,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-11 08:53:34,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-11 08:53:34,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-11 08:53:34,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-11 08:53:34,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-11 08:53:34,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-11 08:53:34,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-11 08:53:34,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-11 08:53:34,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-11 08:53:34,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-11 08:53:34,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-11 08:53:34,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-11 08:53:34,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-11 08:53:34,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-11 08:53:34,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-11 08:53:34,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-11 08:53:34,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-11 08:53:34,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-11 08:53:34,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-11 08:53:34,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-11 08:53:34,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-11 08:53:34,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-11 08:53:34,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-11 08:53:34,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-11 08:53:34,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-11 08:53:34,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-11 08:53:34,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-11 08:53:34,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-11 08:53:34,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-11 08:53:34,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-11 08:53:34,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-11 08:53:34,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-11 08:53:34,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-11 08:53:34,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-11 08:53:34,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-11 08:53:34,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-11 08:53:34,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-11 08:53:34,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-11 08:53:34,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-11 08:53:34,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-11 08:53:34,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-11 08:53:34,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-11 08:53:34,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-11 08:53:34,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-11 08:53:34,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-11 08:53:34,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-11 08:53:34,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-11 08:53:34,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-11 08:53:34,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-11 08:53:34,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-11 08:53:34,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-11 08:53:34,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-11 08:53:34,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-11 08:53:34,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-11 08:53:34,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-11 08:53:34,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-11 08:53:34,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-11 08:53:34,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-11 08:53:34,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-11 08:53:34,495 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005984558423028655); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008926687311226545); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005984558423028655); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=6.666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-11 08:53:34,496 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-11 08:53:34,496 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0010518086469620198, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005984558423028655); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008926687311226545); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005984558423028655); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=6.666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-11 08:53:34,843 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 489 ms to try 5000 different iterations.  Found a solution that moves 267 regions; Going from a computed imbalance of 0.0010518086469620198 to a new imbalance of 2.0825379905082664E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.029666666666666668, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-11 08:53:34,844 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1213954116:83 , srv1880098460:83 , srv1889002841:83 , srv1980755879:83 , srv402044541:83 , srv715736950:83 , srv105233554:84 , srv1066857067:84 , srv1370321424:84 , srv1416718888:84 , srv1422187148:84 , srv1442255124:84 , srv1450205080:84 , srv1527410656:84 , srv1584987150:84 , srv1751610126:84 , srv1835241327:84 , srv1986875100:84 , srv1992914079:84 , srv201487997:84 , srv2017131521:84 , srv2075176166:84 , srv2087195590:84 , srv2091765922:84 , srv2121361670:84 , srv279192051:84 , srv324620804:84 , srv325393181:84 , srv331825714:84 , srv348360542:84 , srv382522915:84 , srv47491821:84 , srv530448304:84 , srv561308370:84 , srv620862355:84 , srv704472988:84 , srv756457572:84 , srv762266303:84 , srv813275997:84 , srv83112443:84 , srv850142140:84 , srv874933324:84 , srv903820536:84 , srv910300102:84 , srv1031328482:85 , srv106571720:85 , srv1076729323:85 , srv1112599708:85 , srv1157265950:85 , srv1174895264:85 , srv1200484306:85 , srv1300121444:85 , srv1340409870:85 , srv1396926753:85 , srv1430903384:85 , srv1442495696:85 , srv1502686865:85 , srv152653220:85 , srv1544057090:85 , srv1561560914:85 , srv1642079494:85 , srv1676708918:85 , srv1707961612:85 , srv1731346562:85 , srv1770204534:85 , srv177588469:85 , srv1805196964:85 , srv1808342035:85 , srv1891142648:85 , srv1946430809:85 , srv1958473037:85 , srv2072218809:85 , srv2134611214:85 , srv228330115:85 , srv248783672:85 , srv251885611:85 , srv343249675:85 , srv358852285:85 , srv389185882:85 , srv413326508:85 , srv47424183:85 , srv49365122:85 , srv495317827:85 , srv496268501:85 , srv526844331:85 , srv577385338:85 , srv588440441:85 , srv612493538:85 , srv632545587:85 , srv66832939:85 , srv763434465:85 , srv816486490:85 , srv870294503:85 , srv906052362:85 , srv925281138:85 , srv1222385931:86 , srv514631293:86 , srv539454120:86 , srv828048214:86 , srv751679753:631 }
2023-05-11 08:53:34,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-11 08:53:34,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1980755879=42, srv2017131521=46, srv496268501=71, srv1066857067=3, srv228330115=53, srv715736950=84, srv539454120=75, srv1422187148=16, srv751679753=85, srv105233554=1, srv402044541=65, srv1544057090=24, srv756457572=86, srv1031328482=0, srv530448304=74, srv828048214=91, srv925281138=99, srv1416718888=15, srv2087195590=49, srv2134611214=52, srv47424183=67, srv1300121444=11, srv612493538=79, srv874933324=95, srv903820536=96, srv152653220=22, srv1112599708=5, srv331825714=59, srv1561560914=25, srv325393181=58, srv816486490=90, srv1676708918=28, srv1835241327=36, srv343249675=60, srv910300102=98, srv763434465=88, srv1174895264=7, srv1880098460=37, srv49365122=69, srv1889002841=38, srv201487997=45, srv251885611=55, srv1213954116=9, srv324620804=57, srv495317827=70, srv2091765922=50, srv248783672=54, srv906052362=97, srv1222385931=10, srv1396926753=14, srv561308370=76, srv1370321424=13, srv389185882=64, srv83112443=92, srv1527410656=23, srv348360542=61, srv358852285=62, srv1442255124=18, srv413326508=66, srv1808342035=35, srv66832939=82, srv279192051=56, srv704472988=83, srv1805196964=34, srv514631293=72, srv813275997=89, srv762266303=87, srv1430903384=17, srv1200484306=8, srv1502686865=21, srv526844331=73, srv47491821=68, srv1958473037=41, srv1076729323=4, srv1642079494=27, srv620862355=80, srv850142140=93, srv1891142648=39, srv577385338=77, srv177588469=33, srv632545587=81, srv870294503=94, srv106571720=2, srv1986875100=43, srv1442495696=19, srv1731346562=30, srv2072218809=47, srv1707961612=29, srv2121361670=51, srv2075176166=48, srv588440441=78, srv1770204534=32, srv1946430809=40, srv1450205080=20, srv1992914079=44, srv1751610126=31, srv382522915=63, srv1157265950=6, srv1340409870=12, srv1584987150=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-11 08:53:34,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-11 08:53:34,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-11 08:53:34,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-11 08:53:34,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-11 08:53:34,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-11 08:53:34,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-11 08:53:34,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-11 08:53:34,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-11 08:53:34,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-11 08:53:34,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-11 08:53:34,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-11 08:53:34,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-11 08:53:34,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-11 08:53:34,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-11 08:53:34,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-11 08:53:34,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-11 08:53:34,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-11 08:53:34,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-11 08:53:34,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-11 08:53:34,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-11 08:53:34,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-11 08:53:34,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-11 08:53:34,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-11 08:53:34,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-11 08:53:34,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-11 08:53:34,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-11 08:53:34,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-11 08:53:34,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-11 08:53:34,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-11 08:53:34,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-11 08:53:34,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-11 08:53:34,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-11 08:53:34,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-11 08:53:34,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-11 08:53:34,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-11 08:53:34,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-11 08:53:34,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-11 08:53:34,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-11 08:53:34,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-11 08:53:34,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-11 08:53:34,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-11 08:53:34,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-11 08:53:34,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-11 08:53:34,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-11 08:53:34,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-11 08:53:34,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-11 08:53:34,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-11 08:53:34,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-11 08:53:34,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-11 08:53:34,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-11 08:53:34,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-11 08:53:34,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-11 08:53:34,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-11 08:53:34,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-11 08:53:34,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-11 08:53:34,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-11 08:53:34,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-11 08:53:34,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-11 08:53:34,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-11 08:53:34,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-11 08:53:34,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-11 08:53:34,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-11 08:53:34,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-11 08:53:34,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-11 08:53:34,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-11 08:53:34,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-11 08:53:34,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-11 08:53:34,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-11 08:53:34,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-11 08:53:34,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-11 08:53:34,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-11 08:53:34,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-11 08:53:34,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-11 08:53:34,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-11 08:53:34,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-11 08:53:34,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-11 08:53:34,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-11 08:53:34,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-11 08:53:34,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-11 08:53:34,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-11 08:53:34,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-11 08:53:34,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-11 08:53:34,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-11 08:53:34,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-11 08:53:34,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-11 08:53:34,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-11 08:53:34,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-11 08:53:34,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-11 08:53:34,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-11 08:53:34,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-11 08:53:34,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-11 08:53:34,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-11 08:53:34,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-11 08:53:34,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-11 08:53:34,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-11 08:53:34,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-11 08:53:34,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-11 08:53:34,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-11 08:53:34,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-11 08:53:34,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-11 08:53:34,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-11 08:53:35,006 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.8955805945877192E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-11 08:53:35,006 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1213954116:83 , srv1880098460:83 , srv1889002841:83 , srv1980755879:83 , srv402044541:83 , srv715736950:83 , srv105233554:84 , srv1066857067:84 , srv1370321424:84 , srv1416718888:84 , srv1422187148:84 , srv1442255124:84 , srv1450205080:84 , srv1527410656:84 , srv1584987150:84 , srv1751610126:84 , srv1835241327:84 , srv1986875100:84 , srv1992914079:84 , srv201487997:84 , srv2017131521:84 , srv2075176166:84 , srv2087195590:84 , srv2091765922:84 , srv2121361670:84 , srv279192051:84 , srv324620804:84 , srv325393181:84 , srv331825714:84 , srv348360542:84 , srv382522915:84 , srv47491821:84 , srv530448304:84 , srv561308370:84 , srv620862355:84 , srv704472988:84 , srv756457572:84 , srv762266303:84 , srv813275997:84 , srv83112443:84 , srv850142140:84 , srv874933324:84 , srv903820536:84 , srv910300102:84 , srv1031328482:85 , srv106571720:85 , srv1076729323:85 , srv1112599708:85 , srv1157265950:85 , srv1174895264:85 , srv1200484306:85 , srv1300121444:85 , srv1340409870:85 , srv1396926753:85 , srv1430903384:85 , srv1442495696:85 , srv1502686865:85 , srv152653220:85 , srv1544057090:85 , srv1561560914:85 , srv1642079494:85 , srv1676708918:85 , srv1707961612:85 , srv1731346562:85 , srv1770204534:85 , srv177588469:85 , srv1805196964:85 , srv1808342035:85 , srv1891142648:85 , srv1946430809:85 , srv1958473037:85 , srv2072218809:85 , srv2134611214:85 , srv228330115:85 , srv248783672:85 , srv251885611:85 , srv343249675:85 , srv358852285:85 , srv389185882:85 , srv413326508:85 , srv47424183:85 , srv49365122:85 , srv495317827:85 , srv496268501:85 , srv526844331:85 , srv577385338:85 , srv588440441:85 , srv612493538:85 , srv632545587:85 , srv66832939:85 , srv763434465:85 , srv816486490:85 , srv870294503:85 , srv906052362:85 , srv925281138:85 , srv1222385931:86 , srv514631293:86 , srv539454120:86 , srv828048214:86 , srv751679753:631 }
2023-05-11 08:53:35,037 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=308 (was 308), ProcessCount=175 (was 175), AvailableMemoryMB=9028 (was 9862)
2023-05-11 08:53:35,047 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=308, ProcessCount=175, AvailableMemoryMB=9026
2023-05-11 08:53:35,047 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-11 08:53:35,047 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-05-11 08:53:35,056 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1161156531:84 , srv1193497101:84 , srv1251820405:84 , srv1451591145:84 , srv1452586509:84 , srv1566750373:84 , srv1843668950:84 , srv1975902132:84 , srv1980316940:84 , srv280614404:84 , srv281714807:84 , srv314494437:84 , srv317690631:84 , srv489852735:84 , srv490475148:84 , srv526721658:84 , srv557021662:84 , srv766933646:84 , srv803595457:84 , srv856182803:84 , srv857521337:84 , srv859447740:84 , srv87372015:84 , srv889250589:84 , srv917242327:84 , srv932870535:84 , srv957256157:84 , srv965492087:84 , srv972290852:84 , srv1191603344:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-05-11 08:53:35,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-11 08:53:35,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv856182803=20, srv917242327=25, srv957256157=27, srv1843668950=7, srv972290852=29, srv1451591145=4, srv1975902132=8, srv1193497101=2, srv1251820405=3, srv490475148=15, srv1980316940=9, srv932870535=26, srv557021662=17, srv314494437=12, srv1566750373=6, srv526721658=16, srv87372015=23, srv280614404=10, srv1191603344=1, srv859447740=22, srv281714807=11, srv1452586509=5, srv1161156531=0, srv766933646=18, srv857521337=21, srv803595457=19, srv317690631=13, srv889250589=24, srv489852735=14, srv965492087=28} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-11 08:53:35,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-11 08:53:35,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-11 08:53:35,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-11 08:53:35,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-11 08:53:35,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-11 08:53:35,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-11 08:53:35,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-11 08:53:35,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-11 08:53:35,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-11 08:53:35,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-11 08:53:35,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-11 08:53:35,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-11 08:53:35,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-11 08:53:35,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-11 08:53:35,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-11 08:53:35,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-11 08:53:35,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-11 08:53:35,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-11 08:53:35,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-11 08:53:35,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-11 08:53:35,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-11 08:53:35,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-11 08:53:35,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-11 08:53:35,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-11 08:53:35,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-11 08:53:35,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-11 08:53:35,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-11 08:53:35,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-11 08:53:35,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-11 08:53:35,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-11 08:53:35,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-05-11 08:53:35,094 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1332523849232659, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-11 08:53:35,094 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912612917178082, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1332523849232659, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-05-11 08:55:35,071 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6266453 different iterations.  Found a solution that moves 1862 regions; Going from a computed imbalance of 0.9912612917178082 to a new imbalance of 4.346000888939571E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6896296296296296, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-11 08:55:35,079 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1161156531:90 , srv1191603344:90 , srv1193497101:90 , srv1251820405:90 , srv1451591145:90 , srv1452586509:90 , srv1566750373:90 , srv1843668950:90 , srv1975902132:90 , srv1980316940:90 , srv280614404:90 , srv281714807:90 , srv314494437:90 , srv317690631:90 , srv489852735:90 , srv490475148:90 , srv526721658:90 , srv557021662:90 , srv766933646:90 , srv803595457:90 , srv856182803:90 , srv857521337:90 , srv859447740:90 , srv87372015:90 , srv889250589:90 , srv917242327:90 , srv932870535:90 , srv957256157:90 , srv965492087:90 , srv972290852:90 }
2023-05-11 08:55:35,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-11 08:55:35,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv856182803=20, srv917242327=25, srv957256157=27, srv1843668950=7, srv972290852=29, srv1451591145=4, srv1975902132=8, srv1193497101=2, srv1251820405=3, srv490475148=15, srv1980316940=9, srv932870535=26, srv557021662=17, srv314494437=12, srv1566750373=6, srv526721658=16, srv87372015=23, srv280614404=10, srv1191603344=1, srv859447740=22, srv281714807=11, srv1452586509=5, srv1161156531=0, srv766933646=18, srv857521337=21, srv803595457=19, srv317690631=13, srv889250589=24, srv489852735=14, srv965492087=28} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-11 08:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-11 08:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-11 08:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-11 08:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-11 08:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-11 08:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-11 08:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-11 08:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-11 08:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-11 08:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-11 08:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-11 08:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-11 08:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-11 08:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-11 08:55:35,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-11 08:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-11 08:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-11 08:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-11 08:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-11 08:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-11 08:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-11 08:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-11 08:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-11 08:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-11 08:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-11 08:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-11 08:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-11 08:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-11 08:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-11 08:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-11 08:55:35,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-05-11 08:55:35,097 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-11 08:55:35,097 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1161156531:90 , srv1191603344:90 , srv1193497101:90 , srv1251820405:90 , srv1451591145:90 , srv1452586509:90 , srv1566750373:90 , srv1843668950:90 , srv1975902132:90 , srv1980316940:90 , srv280614404:90 , srv281714807:90 , srv314494437:90 , srv317690631:90 , srv489852735:90 , srv490475148:90 , srv526721658:90 , srv557021662:90 , srv766933646:90 , srv803595457:90 , srv856182803:90 , srv857521337:90 , srv859447740:90 , srv87372015:90 , srv889250589:90 , srv917242327:90 , srv932870535:90 , srv957256157:90 , srv965492087:90 , srv972290852:90 }
2023-05-11 08:55:35,109 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=179 (was 308), ProcessCount=171 (was 175), AvailableMemoryMB=8895 (was 9026)