2023-04-28 04:53:23,386 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-04-28 04:53:23,682 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-04-28 04:53:23,793 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-04-28 04:53:23,860 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-28 04:53:23,866 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-28 04:53:23,878 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=354, ProcessCount=174, AvailableMemoryMB=11671
2023-04-28 04:53:23,880 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-28 04:53:23,880 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-28 04:53:24,012 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1008477023:84 , srv1067467940:84 , srv1071196675:84 , srv112439873:84 , srv1149186598:84 , srv1155747860:84 , srv1177326205:84 , srv1179004721:84 , srv1220075486:84 , srv1267164202:84 , srv1290074448:84 , srv1293690363:84 , srv1294521391:84 , srv1345476560:84 , srv1348849631:84 , srv1364415175:84 , srv1369451847:84 , srv1442715955:84 , srv1450257050:84 , srv1451563853:84 , srv1457965098:84 , srv1471889439:84 , srv1493865111:84 , srv150379219:84 , srv1522192485:84 , srv1522419839:84 , srv1570091517:84 , srv158801726:84 , srv1598890198:84 , srv1606556607:84 , srv1670754658:84 , srv1679751153:84 , srv1683523247:84 , srv1684074183:84 , srv1686701471:84 , srv1702397225:84 , srv1724776835:84 , srv173134609:84 , srv1736528919:84 , srv1758942497:84 , srv1768528928:84 , srv1785786892:84 , srv1798062646:84 , srv1810028752:84 , srv1814005896:84 , srv1819792784:84 , srv1834852444:84 , srv1846372229:84 , srv193040526:84 , srv1935932983:84 , srv1965177800:84 , srv2008715873:84 , srv201115380:84 , srv2011376955:84 , srv2045121078:84 , srv2059167962:84 , srv2076299241:84 , srv2122519657:84 , srv219867000:84 , srv220779297:84 , srv221858394:84 , srv259059280:84 , srv325960559:84 , srv346557963:84 , srv349891339:84 , srv395940090:84 , srv441767512:84 , srv451418202:84 , srv479877833:84 , srv497801839:84 , srv511015327:84 , srv545001341:84 , srv598487116:84 , srv614836418:84 , srv639705969:84 , srv642169295:84 , srv645027300:84 , srv655448695:84 , srv671652474:84 , srv67284751:84 , srv684329617:84 , srv703567403:84 , srv708205477:84 , srv71360825:84 , srv719274480:84 , srv743183573:84 , srv775732932:84 , srv789438626:84 , srv790075612:84 , srv803969346:84 , srv810582686:84 , srv816423155:84 , srv833233156:84 , srv838069450:84 , srv838959022:84 , srv921738340:84 , srv921868416:84 , srv922511399:84 , srv981401726:84 , srv344089337:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-04-28 04:53:24,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-28 04:53:24,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv219867000=58, srv1155747860=5, srv1570091517=26, srv598487116=73, srv344089337=63, srv1293690363=11, srv201115380=52, srv1683523247=32, srv2076299241=56, srv1834852444=46, srv922511399=98, srv921868416=97, srv684329617=81, srv1965177800=50, srv655448695=78, srv816423155=92, srv1686701471=34, srv346557963=64, srv1679751153=31, srv1702397225=35, srv325960559=62, srv810582686=91, srv921738340=96, srv479877833=69, srv981401726=99, srv1522419839=25, srv545001341=72, srv220779297=59, srv1785786892=41, srv150379219=23, srv1451563853=19, srv1369451847=16, srv1493865111=22, srv1606556607=29, srv1736528919=38, srv743183573=86, srv639705969=75, srv1758942497=39, srv1179004721=7, srv703567403=82, srv1810028752=43, srv838069450=94, srv173134609=37, srv803969346=90, srv2045121078=54, srv1724776835=36, srv2011376955=53, srv789438626=88, srv1598890198=28, srv1345476560=13, srv838959022=95, srv158801726=27, srv349891339=65, srv1935932983=49, srv719274480=85, srv1220075486=8, srv790075612=89, srv1008477023=0, srv1267164202=9, srv1768528928=40, srv221858394=60, srv1177326205=6, srv1798062646=42, srv1846372229=47, srv441767512=67, srv1290074448=10, srv614836418=74, srv645027300=77, srv1348849631=14, srv833233156=93, srv511015327=71, srv1814005896=44, srv451418202=68, srv1670754658=30, srv1819792784=45, srv193040526=48, srv1149186598=4, srv259059280=61, srv1067467940=1, srv775732932=87, srv1442715955=17, srv497801839=70, srv395940090=66, srv1457965098=20, srv2122519657=57, srv1364415175=15, srv1450257050=18, srv2008715873=51, srv671652474=79, srv642169295=76, srv1522192485=24, srv1071196675=2, srv67284751=80, srv708205477=83, srv1294521391=12, srv1684074183=33, srv112439873=3, srv71360825=84, srv2059167962=55, srv1471889439=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-28 04:53:24,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-28 04:53:24,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-28 04:53:24,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-28 04:53:24,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-28 04:53:24,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-28 04:53:24,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-28 04:53:24,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-28 04:53:24,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-28 04:53:24,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-28 04:53:24,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-28 04:53:24,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-28 04:53:24,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-28 04:53:24,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-28 04:53:24,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-28 04:53:24,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-28 04:53:24,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-28 04:53:24,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-28 04:53:24,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-28 04:53:24,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-28 04:53:24,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-28 04:53:24,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-28 04:53:24,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-28 04:53:24,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-28 04:53:24,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-28 04:53:24,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-28 04:53:24,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-28 04:53:24,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-28 04:53:24,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-28 04:53:24,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-28 04:53:24,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-28 04:53:24,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-28 04:53:24,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-28 04:53:24,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-28 04:53:24,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-28 04:53:24,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-28 04:53:24,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-28 04:53:24,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-28 04:53:24,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-28 04:53:24,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-28 04:53:24,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-28 04:53:24,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-28 04:53:24,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-28 04:53:24,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-28 04:53:24,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-28 04:53:24,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-28 04:53:24,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-28 04:53:24,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-28 04:53:24,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-28 04:53:24,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-28 04:53:24,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-28 04:53:24,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-28 04:53:24,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-28 04:53:24,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-28 04:53:24,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-28 04:53:24,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-28 04:53:24,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-28 04:53:24,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-28 04:53:24,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-28 04:53:24,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-28 04:53:24,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-28 04:53:24,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-28 04:53:24,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-28 04:53:24,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-28 04:53:24,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-28 04:53:24,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-28 04:53:24,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-28 04:53:24,145 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-28 04:53:24,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-28 04:53:24,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-28 04:53:24,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-28 04:53:24,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-28 04:53:24,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-28 04:53:24,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-28 04:53:24,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-28 04:53:24,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-28 04:53:24,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-28 04:53:24,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-28 04:53:24,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-28 04:53:24,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-28 04:53:24,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-28 04:53:24,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-28 04:53:24,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-28 04:53:24,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-28 04:53:24,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-28 04:53:24,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-28 04:53:24,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-28 04:53:24,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-28 04:53:24,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-28 04:53:24,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-28 04:53:24,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-28 04:53:24,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-28 04:53:24,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-28 04:53:24,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-28 04:53:24,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-28 04:53:24,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-28 04:53:24,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-28 04:53:24,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-28 04:53:24,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-28 04:53:24,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-28 04:53:24,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-28 04:53:24,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-28 04:53:24,472 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-28 04:53:24,473 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-28 04:53:24,473 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-28 04:53:25,036 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 853 ms to try 5000 different iterations.  Found a solution that moves 4708 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18201835462453794. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04064512808703673, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053625043984243115, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5231111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04064512808703673, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17766666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23991666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-28 04:53:25,048 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv545001341:81 , srv708205477:81 , srv1442715955:82 , srv1785786892:82 , srv2076299241:82 , srv703567403:82 , srv719274480:82 , srv743183573:82 , srv2045121078:83 , srv221858394:83 , srv259059280:83 , srv614836418:83 , srv642169295:83 , srv1149186598:84 , srv1155747860:84 , srv1177326205:84 , srv1290074448:84 , srv1451563853:84 , srv1471889439:84 , srv1522192485:84 , srv1670754658:84 , srv173134609:84 , srv1935932983:84 , srv325960559:84 , srv598487116:84 , srv671652474:84 , srv684329617:84 , srv833233156:84 , srv838069450:84 , srv112439873:85 , srv1493865111:85 , srv1522419839:85 , srv158801726:85 , srv1686701471:85 , srv193040526:85 , srv2008715873:85 , srv201115380:85 , srv67284751:85 , srv1067467940:86 , srv1071196675:86 , srv1345476560:86 , srv1348849631:86 , srv150379219:86 , srv1598890198:86 , srv1606556607:86 , srv1684074183:86 , srv1736528919:86 , srv1798062646:86 , srv2122519657:86 , srv220779297:86 , srv346557963:86 , srv395940090:86 , srv479877833:86 , srv645027300:86 , srv655448695:86 , srv790075612:86 , srv810582686:86 , srv921738340:86 , srv921868416:86 , srv922511399:86 , srv1220075486:87 , srv1267164202:87 , srv1450257050:87 , srv1457965098:87 , srv1570091517:87 , srv1814005896:87 , srv349891339:87 , srv441767512:87 , srv451418202:87 , srv816423155:87 , srv1179004721:88 , srv1294521391:88 , srv1768528928:88 , srv2011376955:88 , srv2059167962:88 , srv219867000:88 , srv497801839:88 , srv803969346:88 , srv1008477023:89 , srv1683523247:89 , srv1758942497:89 , srv1834852444:89 , srv1846372229:89 , srv71360825:89 , srv789438626:89 , srv981401726:89 , srv1702397225:90 , srv1965177800:90 , srv639705969:90 , srv775732932:90 , srv1293690363:91 , srv1369451847:91 , srv511015327:91 , srv1364415175:92 , srv838959022:92 , srv1679751153:93 , srv1724776835:93 , srv1810028752:94 , srv1819792784:96 , srv344089337:451 }
2023-04-28 04:53:25,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-28 04:53:25,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv219867000=58, srv1155747860=5, srv1570091517=26, srv598487116=73, srv344089337=63, srv1293690363=11, srv201115380=52, srv1683523247=32, srv2076299241=56, srv1834852444=46, srv922511399=98, srv921868416=97, srv684329617=81, srv1965177800=50, srv655448695=78, srv816423155=92, srv1686701471=34, srv346557963=64, srv1679751153=31, srv1702397225=35, srv325960559=62, srv810582686=91, srv921738340=96, srv479877833=69, srv981401726=99, srv1522419839=25, srv545001341=72, srv220779297=59, srv1785786892=41, srv150379219=23, srv1451563853=19, srv1369451847=16, srv1493865111=22, srv1606556607=29, srv1736528919=38, srv743183573=86, srv639705969=75, srv1758942497=39, srv1179004721=7, srv703567403=82, srv1810028752=43, srv838069450=94, srv173134609=37, srv803969346=90, srv2045121078=54, srv1724776835=36, srv2011376955=53, srv789438626=88, srv1598890198=28, srv1345476560=13, srv838959022=95, srv158801726=27, srv349891339=65, srv1935932983=49, srv719274480=85, srv1220075486=8, srv790075612=89, srv1008477023=0, srv1267164202=9, srv1768528928=40, srv221858394=60, srv1177326205=6, srv1798062646=42, srv1846372229=47, srv441767512=67, srv1290074448=10, srv614836418=74, srv645027300=77, srv1348849631=14, srv833233156=93, srv511015327=71, srv1814005896=44, srv451418202=68, srv1670754658=30, srv1819792784=45, srv193040526=48, srv1149186598=4, srv259059280=61, srv1067467940=1, srv775732932=87, srv1442715955=17, srv497801839=70, srv395940090=66, srv1457965098=20, srv2122519657=57, srv1364415175=15, srv1450257050=18, srv2008715873=51, srv671652474=79, srv642169295=76, srv1522192485=24, srv1071196675=2, srv67284751=80, srv708205477=83, srv1294521391=12, srv1684074183=33, srv112439873=3, srv71360825=84, srv2059167962=55, srv1471889439=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-28 04:53:25,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-28 04:53:25,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-28 04:53:25,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-28 04:53:25,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-28 04:53:25,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-28 04:53:25,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-28 04:53:25,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-28 04:53:25,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-28 04:53:25,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-28 04:53:25,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-28 04:53:25,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-28 04:53:25,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-28 04:53:25,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-28 04:53:25,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-28 04:53:25,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-28 04:53:25,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-28 04:53:25,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-28 04:53:25,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-28 04:53:25,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-28 04:53:25,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-28 04:53:25,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-28 04:53:25,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-28 04:53:25,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-28 04:53:25,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-28 04:53:25,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-28 04:53:25,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-28 04:53:25,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-28 04:53:25,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-28 04:53:25,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-28 04:53:25,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-28 04:53:25,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-28 04:53:25,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-28 04:53:25,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-28 04:53:25,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-28 04:53:25,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-28 04:53:25,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-28 04:53:25,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-28 04:53:25,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-28 04:53:25,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-28 04:53:25,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-28 04:53:25,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-28 04:53:25,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-28 04:53:25,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-28 04:53:25,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-28 04:53:25,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-28 04:53:25,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-28 04:53:25,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-28 04:53:25,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-28 04:53:25,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-28 04:53:25,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-28 04:53:25,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-28 04:53:25,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-28 04:53:25,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-28 04:53:25,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-28 04:53:25,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-28 04:53:25,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-28 04:53:25,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-28 04:53:25,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-28 04:53:25,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-28 04:53:25,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-28 04:53:25,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-28 04:53:25,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-28 04:53:25,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-28 04:53:25,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-28 04:53:25,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-28 04:53:25,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-28 04:53:25,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-28 04:53:25,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-28 04:53:25,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-28 04:53:25,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-28 04:53:25,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-28 04:53:25,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-28 04:53:25,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-28 04:53:25,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-28 04:53:25,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-28 04:53:25,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-28 04:53:25,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-28 04:53:25,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-28 04:53:25,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-28 04:53:25,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-28 04:53:25,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-28 04:53:25,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-28 04:53:25,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-28 04:53:25,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-28 04:53:25,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-28 04:53:25,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-28 04:53:25,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-28 04:53:25,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-28 04:53:25,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-28 04:53:25,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-28 04:53:25,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-28 04:53:25,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-28 04:53:25,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-28 04:53:25,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-28 04:53:25,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-28 04:53:25,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-28 04:53:25,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-28 04:53:25,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-28 04:53:25,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-28 04:53:25,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-28 04:53:25,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-28 04:53:25,191 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04064512808703673, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053625043984243115, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04064512808703673, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17766666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23991666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-28 04:53:25,191 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-28 04:53:25,192 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1819853885039389, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04064512808703673, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053625043984243115, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04064512808703673, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17766666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23991666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-28 04:53:25,387 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 286 ms to try 5000 different iterations.  Found a solution that moves 2630 regions; Going from a computed imbalance of 0.1819853885039389 to a new imbalance of 0.013437102382151487. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02001807388848297); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031262259099359896, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2922222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02001807388848297); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0105); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.041416666666666664, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-28 04:53:25,394 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1819792784:74 , srv838959022:78 , srv1364415175:79 , srv1679751153:79 , srv1724776835:79 , srv1369451847:81 , srv1810028752:81 , srv1683523247:82 , srv1965177800:82 , srv497801839:82 , srv639705969:82 , srv981401726:82 , srv1008477023:83 , srv1179004721:83 , srv1293690363:83 , srv1294521391:83 , srv1758942497:83 , srv1768528928:83 , srv2059167962:83 , srv775732932:83 , srv789438626:83 , srv1071196675:84 , srv1702397225:84 , srv1846372229:84 , srv2011376955:84 , srv511015327:84 , srv803969346:84 , srv1450257050:85 , srv1736528919:85 , srv1834852444:85 , srv2122519657:85 , srv219867000:85 , srv441767512:85 , srv71360825:85 , srv790075612:85 , srv810582686:85 , srv1067467940:86 , srv1220075486:86 , srv1345476560:86 , srv1457965098:86 , srv1493865111:86 , srv150379219:86 , srv158801726:86 , srv1598890198:86 , srv1606556607:86 , srv1684074183:86 , srv1814005896:86 , srv2008715873:86 , srv220779297:86 , srv346557963:86 , srv349891339:86 , srv395940090:86 , srv451418202:86 , srv655448695:86 , srv922511399:86 , srv112439873:87 , srv1177326205:87 , srv1267164202:87 , srv1348849631:87 , srv1451563853:87 , srv1522419839:87 , srv1570091517:87 , srv1798062646:87 , srv193040526:87 , srv325960559:87 , srv479877833:87 , srv67284751:87 , srv684329617:87 , srv833233156:87 , srv921738340:87 , srv1149186598:88 , srv1471889439:88 , srv1522192485:88 , srv1686701471:88 , srv173134609:88 , srv598487116:88 , srv614836418:88 , srv645027300:88 , srv671652474:88 , srv719274480:88 , srv816423155:88 , srv838069450:88 , srv921868416:88 , srv1290074448:89 , srv1935932983:89 , srv201115380:89 , srv2045121078:89 , srv221858394:89 , srv1155747860:90 , srv1442715955:90 , srv1670754658:90 , srv1785786892:90 , srv2076299241:90 , srv259059280:90 , srv708205477:90 , srv545001341:91 , srv642169295:91 , srv703567403:91 , srv743183573:91 , srv344089337:501 }
2023-04-28 04:53:25,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-28 04:53:25,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv219867000=58, srv1155747860=5, srv1570091517=26, srv598487116=73, srv344089337=63, srv1293690363=11, srv201115380=52, srv1683523247=32, srv2076299241=56, srv1834852444=46, srv922511399=98, srv921868416=97, srv684329617=81, srv1965177800=50, srv655448695=78, srv816423155=92, srv1686701471=34, srv346557963=64, srv1679751153=31, srv1702397225=35, srv325960559=62, srv810582686=91, srv921738340=96, srv479877833=69, srv981401726=99, srv1522419839=25, srv545001341=72, srv220779297=59, srv1785786892=41, srv150379219=23, srv1451563853=19, srv1369451847=16, srv1493865111=22, srv1606556607=29, srv1736528919=38, srv743183573=86, srv639705969=75, srv1758942497=39, srv1179004721=7, srv703567403=82, srv1810028752=43, srv838069450=94, srv173134609=37, srv803969346=90, srv2045121078=54, srv1724776835=36, srv2011376955=53, srv789438626=88, srv1598890198=28, srv1345476560=13, srv838959022=95, srv158801726=27, srv349891339=65, srv1935932983=49, srv719274480=85, srv1220075486=8, srv790075612=89, srv1008477023=0, srv1267164202=9, srv1768528928=40, srv221858394=60, srv1177326205=6, srv1798062646=42, srv1846372229=47, srv441767512=67, srv1290074448=10, srv614836418=74, srv645027300=77, srv1348849631=14, srv833233156=93, srv511015327=71, srv1814005896=44, srv451418202=68, srv1670754658=30, srv1819792784=45, srv193040526=48, srv1149186598=4, srv259059280=61, srv1067467940=1, srv775732932=87, srv1442715955=17, srv497801839=70, srv395940090=66, srv1457965098=20, srv2122519657=57, srv1364415175=15, srv1450257050=18, srv2008715873=51, srv671652474=79, srv642169295=76, srv1522192485=24, srv1071196675=2, srv67284751=80, srv708205477=83, srv1294521391=12, srv1684074183=33, srv112439873=3, srv71360825=84, srv2059167962=55, srv1471889439=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-28 04:53:25,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-28 04:53:25,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-28 04:53:25,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-28 04:53:25,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-28 04:53:25,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-28 04:53:25,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-28 04:53:25,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-28 04:53:25,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-28 04:53:25,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-28 04:53:25,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-28 04:53:25,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-28 04:53:25,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-28 04:53:25,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-28 04:53:25,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-28 04:53:25,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-28 04:53:25,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-28 04:53:25,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-28 04:53:25,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-28 04:53:25,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-28 04:53:25,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-28 04:53:25,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-28 04:53:25,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-28 04:53:25,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-28 04:53:25,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-28 04:53:25,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-28 04:53:25,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-28 04:53:25,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-28 04:53:25,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-28 04:53:25,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-28 04:53:25,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-28 04:53:25,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-28 04:53:25,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-28 04:53:25,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-28 04:53:25,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-28 04:53:25,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-28 04:53:25,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-28 04:53:25,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-28 04:53:25,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-28 04:53:25,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-28 04:53:25,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-28 04:53:25,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-28 04:53:25,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-28 04:53:25,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-28 04:53:25,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-28 04:53:25,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-28 04:53:25,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-28 04:53:25,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-28 04:53:25,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-28 04:53:25,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-28 04:53:25,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-28 04:53:25,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-28 04:53:25,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-28 04:53:25,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-28 04:53:25,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-28 04:53:25,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-28 04:53:25,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-28 04:53:25,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-28 04:53:25,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-28 04:53:25,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-28 04:53:25,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-28 04:53:25,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-28 04:53:25,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-28 04:53:25,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-28 04:53:25,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-28 04:53:25,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-28 04:53:25,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-28 04:53:25,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-28 04:53:25,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-28 04:53:25,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-28 04:53:25,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-28 04:53:25,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-28 04:53:25,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-28 04:53:25,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-28 04:53:25,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-28 04:53:25,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-28 04:53:25,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-28 04:53:25,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-28 04:53:25,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-28 04:53:25,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-28 04:53:25,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-28 04:53:25,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-28 04:53:25,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-28 04:53:25,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-28 04:53:25,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-28 04:53:25,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-28 04:53:25,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-28 04:53:25,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-28 04:53:25,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-28 04:53:25,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-28 04:53:25,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-28 04:53:25,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-28 04:53:25,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-28 04:53:25,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-28 04:53:25,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-28 04:53:25,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-28 04:53:25,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-28 04:53:25,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-28 04:53:25,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-28 04:53:25,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-28 04:53:25,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-28 04:53:25,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-28 04:53:25,508 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02001807388848297); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031262259099359896, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02001807388848297); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0105); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.041416666666666664, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-28 04:53:25,508 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-28 04:53:25,508 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013418686728545827, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02001807388848297); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031262259099359896, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02001807388848297); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0105); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.041416666666666664, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-28 04:53:25,710 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 275 ms to try 5000 different iterations.  Found a solution that moves 986 regions; Going from a computed imbalance of 0.013418686728545827 to a new imbalance of 7.63750829731869E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054318436834303405); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007324451309205374); MoveCostFunction : (multiplier=7.0, imbalance=0.10955555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0054318436834303405); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-28 04:53:25,711 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1810028752:82 , srv816423155:82 , srv1293690363:83 , srv511015327:83 , srv642169295:83 , srv645027300:83 , srv71360825:83 , srv1155747860:84 , srv1220075486:84 , srv1267164202:84 , srv1348849631:84 , srv1457965098:84 , srv1570091517:84 , srv1686701471:84 , srv1702397225:84 , srv1814005896:84 , srv1834852444:84 , srv1846372229:84 , srv2076299241:84 , srv921868416:84 , srv1008477023:85 , srv112439873:85 , srv1149186598:85 , srv1290074448:85 , srv1450257050:85 , srv1471889439:85 , srv150379219:85 , srv1522192485:85 , srv1522419839:85 , srv1598890198:85 , srv1606556607:85 , srv1670754658:85 , srv1724776835:85 , srv173134609:85 , srv1798062646:85 , srv193040526:85 , srv1935932983:85 , srv1965177800:85 , srv201115380:85 , srv2045121078:85 , srv219867000:85 , srv220779297:85 , srv221858394:85 , srv349891339:85 , srv395940090:85 , srv441767512:85 , srv451418202:85 , srv545001341:85 , srv598487116:85 , srv639705969:85 , srv655448695:85 , srv67284751:85 , srv703567403:85 , srv743183573:85 , srv775732932:85 , srv803969346:85 , srv810582686:85 , srv921738340:85 , srv1067467940:86 , srv1345476560:86 , srv1369451847:86 , srv1442715955:86 , srv1493865111:86 , srv158801726:86 , srv1679751153:86 , srv1684074183:86 , srv1736528919:86 , srv1758942497:86 , srv1768528928:86 , srv1785786892:86 , srv1819792784:86 , srv2011376955:86 , srv2059167962:86 , srv259059280:86 , srv325960559:86 , srv346557963:86 , srv479877833:86 , srv614836418:86 , srv671652474:86 , srv789438626:86 , srv838069450:86 , srv922511399:86 , srv981401726:86 , srv1177326205:87 , srv1179004721:87 , srv1294521391:87 , srv1364415175:87 , srv1451563853:87 , srv1683523247:87 , srv2008715873:87 , srv2122519657:87 , srv684329617:87 , srv708205477:87 , srv719274480:87 , srv790075612:87 , srv833233156:87 , srv1071196675:88 , srv497801839:88 , srv838959022:88 , srv344089337:554 }
2023-04-28 04:53:25,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-28 04:53:25,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv219867000=58, srv1155747860=5, srv1570091517=26, srv598487116=73, srv344089337=63, srv1293690363=11, srv201115380=52, srv1683523247=32, srv2076299241=56, srv1834852444=46, srv922511399=98, srv921868416=97, srv684329617=81, srv1965177800=50, srv655448695=78, srv816423155=92, srv1686701471=34, srv346557963=64, srv1679751153=31, srv1702397225=35, srv325960559=62, srv810582686=91, srv921738340=96, srv479877833=69, srv981401726=99, srv1522419839=25, srv545001341=72, srv220779297=59, srv1785786892=41, srv150379219=23, srv1451563853=19, srv1369451847=16, srv1493865111=22, srv1606556607=29, srv1736528919=38, srv743183573=86, srv639705969=75, srv1758942497=39, srv1179004721=7, srv703567403=82, srv1810028752=43, srv838069450=94, srv173134609=37, srv803969346=90, srv2045121078=54, srv1724776835=36, srv2011376955=53, srv789438626=88, srv1598890198=28, srv1345476560=13, srv838959022=95, srv158801726=27, srv349891339=65, srv1935932983=49, srv719274480=85, srv1220075486=8, srv790075612=89, srv1008477023=0, srv1267164202=9, srv1768528928=40, srv221858394=60, srv1177326205=6, srv1798062646=42, srv1846372229=47, srv441767512=67, srv1290074448=10, srv614836418=74, srv645027300=77, srv1348849631=14, srv833233156=93, srv511015327=71, srv1814005896=44, srv451418202=68, srv1670754658=30, srv1819792784=45, srv193040526=48, srv1149186598=4, srv259059280=61, srv1067467940=1, srv775732932=87, srv1442715955=17, srv497801839=70, srv395940090=66, srv1457965098=20, srv2122519657=57, srv1364415175=15, srv1450257050=18, srv2008715873=51, srv671652474=79, srv642169295=76, srv1522192485=24, srv1071196675=2, srv67284751=80, srv708205477=83, srv1294521391=12, srv1684074183=33, srv112439873=3, srv71360825=84, srv2059167962=55, srv1471889439=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-28 04:53:25,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-28 04:53:25,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-28 04:53:25,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-28 04:53:25,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-28 04:53:25,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-28 04:53:25,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-28 04:53:25,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-28 04:53:25,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-28 04:53:25,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-28 04:53:25,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-28 04:53:25,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-28 04:53:25,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-28 04:53:25,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-28 04:53:25,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-28 04:53:25,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-28 04:53:25,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-28 04:53:25,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-28 04:53:25,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-28 04:53:25,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-28 04:53:25,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-28 04:53:25,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-28 04:53:25,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-28 04:53:25,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-28 04:53:25,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-28 04:53:25,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-28 04:53:25,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-28 04:53:25,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-28 04:53:25,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-28 04:53:25,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-28 04:53:25,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-28 04:53:25,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-28 04:53:25,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-28 04:53:25,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-28 04:53:25,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-28 04:53:25,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-28 04:53:25,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-28 04:53:25,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-28 04:53:25,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-28 04:53:25,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-28 04:53:25,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-28 04:53:25,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-28 04:53:25,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-28 04:53:25,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-28 04:53:25,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-28 04:53:25,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-28 04:53:25,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-28 04:53:25,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-28 04:53:25,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-28 04:53:25,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-28 04:53:25,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-28 04:53:25,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-28 04:53:25,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-28 04:53:25,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-28 04:53:25,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-28 04:53:25,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-28 04:53:25,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-28 04:53:25,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-28 04:53:25,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-28 04:53:25,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-28 04:53:25,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-28 04:53:25,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-28 04:53:25,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-28 04:53:25,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-28 04:53:25,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-28 04:53:25,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-28 04:53:25,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-28 04:53:25,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-28 04:53:25,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-28 04:53:25,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-28 04:53:25,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-28 04:53:25,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-28 04:53:25,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-28 04:53:25,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-28 04:53:25,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-28 04:53:25,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-28 04:53:25,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-28 04:53:25,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-28 04:53:25,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-28 04:53:25,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-28 04:53:25,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-28 04:53:25,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-28 04:53:25,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-28 04:53:25,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-28 04:53:25,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-28 04:53:25,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-28 04:53:25,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-28 04:53:25,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-28 04:53:25,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-28 04:53:25,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-28 04:53:25,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-28 04:53:25,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-28 04:53:25,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-28 04:53:25,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-28 04:53:25,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-28 04:53:25,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-28 04:53:25,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-28 04:53:25,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-28 04:53:25,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-28 04:53:25,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-28 04:53:25,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-28 04:53:25,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-28 04:53:25,822 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054318436834303405); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007324451309205374); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0054318436834303405); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-28 04:53:25,822 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-28 04:53:25,822 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.568467101671626E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054318436834303405); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007324451309205374); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0054318436834303405); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-28 04:53:26,077 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 336 ms to try 5000 different iterations.  Found a solution that moves 224 regions; Going from a computed imbalance of 7.568467101671626E-4 to a new imbalance of 1.2767056392346672E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.024888888888888887); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-28 04:53:26,078 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1442715955:83 , srv1670754658:83 , srv2008715873:83 , srv201115380:83 , srv259059280:83 , srv479877833:83 , srv708205477:83 , srv775732932:83 , srv1067467940:84 , srv1155747860:84 , srv1177326205:84 , srv1179004721:84 , srv1294521391:84 , srv1364415175:84 , srv1369451847:84 , srv1451563853:84 , srv150379219:84 , srv1679751153:84 , srv1683523247:84 , srv1684074183:84 , srv1702397225:84 , srv1758942497:84 , srv1768528928:84 , srv1785786892:84 , srv1798062646:84 , srv1810028752:84 , srv1814005896:84 , srv1834852444:84 , srv1935932983:84 , srv2011376955:84 , srv219867000:84 , srv220779297:84 , srv221858394:84 , srv346557963:84 , srv349891339:84 , srv451418202:84 , srv497801839:84 , srv511015327:84 , srv671652474:84 , srv684329617:84 , srv743183573:84 , srv789438626:84 , srv790075612:84 , srv833233156:84 , srv838069450:84 , srv838959022:84 , srv921738340:84 , srv921868416:84 , srv1008477023:85 , srv1071196675:85 , srv112439873:85 , srv1149186598:85 , srv1220075486:85 , srv1267164202:85 , srv1290074448:85 , srv1293690363:85 , srv1345476560:85 , srv1348849631:85 , srv1450257050:85 , srv1457965098:85 , srv1471889439:85 , srv1493865111:85 , srv1522192485:85 , srv1522419839:85 , srv1570091517:85 , srv158801726:85 , srv1598890198:85 , srv1606556607:85 , srv1686701471:85 , srv1724776835:85 , srv173134609:85 , srv1736528919:85 , srv1846372229:85 , srv193040526:85 , srv1965177800:85 , srv2045121078:85 , srv2059167962:85 , srv2122519657:85 , srv395940090:85 , srv441767512:85 , srv545001341:85 , srv598487116:85 , srv614836418:85 , srv639705969:85 , srv642169295:85 , srv645027300:85 , srv655448695:85 , srv67284751:85 , srv703567403:85 , srv71360825:85 , srv719274480:85 , srv803969346:85 , srv816423155:85 , srv922511399:85 , srv981401726:85 , srv1819792784:86 , srv2076299241:86 , srv325960559:86 , srv810582686:86 , srv344089337:637 }
2023-04-28 04:53:26,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-28 04:53:26,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv219867000=58, srv1155747860=5, srv1570091517=26, srv598487116=73, srv344089337=63, srv1293690363=11, srv201115380=52, srv1683523247=32, srv2076299241=56, srv1834852444=46, srv922511399=98, srv921868416=97, srv684329617=81, srv1965177800=50, srv655448695=78, srv816423155=92, srv1686701471=34, srv346557963=64, srv1679751153=31, srv1702397225=35, srv325960559=62, srv810582686=91, srv921738340=96, srv479877833=69, srv981401726=99, srv1522419839=25, srv545001341=72, srv220779297=59, srv1785786892=41, srv150379219=23, srv1451563853=19, srv1369451847=16, srv1493865111=22, srv1606556607=29, srv1736528919=38, srv743183573=86, srv639705969=75, srv1758942497=39, srv1179004721=7, srv703567403=82, srv1810028752=43, srv838069450=94, srv173134609=37, srv803969346=90, srv2045121078=54, srv1724776835=36, srv2011376955=53, srv789438626=88, srv1598890198=28, srv1345476560=13, srv838959022=95, srv158801726=27, srv349891339=65, srv1935932983=49, srv719274480=85, srv1220075486=8, srv790075612=89, srv1008477023=0, srv1267164202=9, srv1768528928=40, srv221858394=60, srv1177326205=6, srv1798062646=42, srv1846372229=47, srv441767512=67, srv1290074448=10, srv614836418=74, srv645027300=77, srv1348849631=14, srv833233156=93, srv511015327=71, srv1814005896=44, srv451418202=68, srv1670754658=30, srv1819792784=45, srv193040526=48, srv1149186598=4, srv259059280=61, srv1067467940=1, srv775732932=87, srv1442715955=17, srv497801839=70, srv395940090=66, srv1457965098=20, srv2122519657=57, srv1364415175=15, srv1450257050=18, srv2008715873=51, srv671652474=79, srv642169295=76, srv1522192485=24, srv1071196675=2, srv67284751=80, srv708205477=83, srv1294521391=12, srv1684074183=33, srv112439873=3, srv71360825=84, srv2059167962=55, srv1471889439=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-28 04:53:26,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-28 04:53:26,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-28 04:53:26,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-28 04:53:26,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-28 04:53:26,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-28 04:53:26,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-28 04:53:26,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-28 04:53:26,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-28 04:53:26,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-28 04:53:26,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-28 04:53:26,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-28 04:53:26,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-28 04:53:26,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-28 04:53:26,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-28 04:53:26,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-28 04:53:26,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-28 04:53:26,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-28 04:53:26,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-28 04:53:26,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-28 04:53:26,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-28 04:53:26,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-28 04:53:26,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-28 04:53:26,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-28 04:53:26,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-28 04:53:26,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-28 04:53:26,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-28 04:53:26,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-28 04:53:26,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-28 04:53:26,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-28 04:53:26,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-28 04:53:26,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-28 04:53:26,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-28 04:53:26,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-28 04:53:26,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-28 04:53:26,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-28 04:53:26,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-28 04:53:26,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-28 04:53:26,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-28 04:53:26,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-28 04:53:26,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-28 04:53:26,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-28 04:53:26,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-28 04:53:26,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-28 04:53:26,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-28 04:53:26,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-28 04:53:26,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-28 04:53:26,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-28 04:53:26,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-28 04:53:26,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-28 04:53:26,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-28 04:53:26,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-28 04:53:26,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-28 04:53:26,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-28 04:53:26,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-28 04:53:26,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-28 04:53:26,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-28 04:53:26,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-28 04:53:26,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-28 04:53:26,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-28 04:53:26,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-28 04:53:26,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-28 04:53:26,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-28 04:53:26,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-28 04:53:26,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-28 04:53:26,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-28 04:53:26,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-28 04:53:26,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-28 04:53:26,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-28 04:53:26,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-28 04:53:26,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-28 04:53:26,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-28 04:53:26,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-28 04:53:26,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-28 04:53:26,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-28 04:53:26,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-28 04:53:26,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-28 04:53:26,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-28 04:53:26,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-28 04:53:26,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-28 04:53:26,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-28 04:53:26,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-28 04:53:26,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-28 04:53:26,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-28 04:53:26,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-28 04:53:26,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-28 04:53:26,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-28 04:53:26,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-28 04:53:26,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-28 04:53:26,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-28 04:53:26,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-28 04:53:26,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-28 04:53:26,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-28 04:53:26,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-28 04:53:26,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-28 04:53:26,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-28 04:53:26,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-28 04:53:26,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-28 04:53:26,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-28 04:53:26,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-28 04:53:26,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-28 04:53:26,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-28 04:53:26,191 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.1198574868518857E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-28 04:53:26,191 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1442715955:83 , srv1670754658:83 , srv2008715873:83 , srv201115380:83 , srv259059280:83 , srv479877833:83 , srv708205477:83 , srv775732932:83 , srv1067467940:84 , srv1155747860:84 , srv1177326205:84 , srv1179004721:84 , srv1294521391:84 , srv1364415175:84 , srv1369451847:84 , srv1451563853:84 , srv150379219:84 , srv1679751153:84 , srv1683523247:84 , srv1684074183:84 , srv1702397225:84 , srv1758942497:84 , srv1768528928:84 , srv1785786892:84 , srv1798062646:84 , srv1810028752:84 , srv1814005896:84 , srv1834852444:84 , srv1935932983:84 , srv2011376955:84 , srv219867000:84 , srv220779297:84 , srv221858394:84 , srv346557963:84 , srv349891339:84 , srv451418202:84 , srv497801839:84 , srv511015327:84 , srv671652474:84 , srv684329617:84 , srv743183573:84 , srv789438626:84 , srv790075612:84 , srv833233156:84 , srv838069450:84 , srv838959022:84 , srv921738340:84 , srv921868416:84 , srv1008477023:85 , srv1071196675:85 , srv112439873:85 , srv1149186598:85 , srv1220075486:85 , srv1267164202:85 , srv1290074448:85 , srv1293690363:85 , srv1345476560:85 , srv1348849631:85 , srv1450257050:85 , srv1457965098:85 , srv1471889439:85 , srv1493865111:85 , srv1522192485:85 , srv1522419839:85 , srv1570091517:85 , srv158801726:85 , srv1598890198:85 , srv1606556607:85 , srv1686701471:85 , srv1724776835:85 , srv173134609:85 , srv1736528919:85 , srv1846372229:85 , srv193040526:85 , srv1965177800:85 , srv2045121078:85 , srv2059167962:85 , srv2122519657:85 , srv395940090:85 , srv441767512:85 , srv545001341:85 , srv598487116:85 , srv614836418:85 , srv639705969:85 , srv642169295:85 , srv645027300:85 , srv655448695:85 , srv67284751:85 , srv703567403:85 , srv71360825:85 , srv719274480:85 , srv803969346:85 , srv816423155:85 , srv922511399:85 , srv981401726:85 , srv1819792784:86 , srv2076299241:86 , srv325960559:86 , srv810582686:86 , srv344089337:637 }
2023-04-28 04:53:26,215 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=354 (was 354), ProcessCount=174 (was 174), AvailableMemoryMB=11010 (was 11671)
2023-04-28 04:53:26,221 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=354, ProcessCount=174, AvailableMemoryMB=11010
2023-04-28 04:53:26,222 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-28 04:53:26,222 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-04-28 04:53:26,229 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1117860974:84 , srv114232896:84 , srv1150614122:84 , srv1275809153:84 , srv1358209742:84 , srv1376234994:84 , srv1519099097:84 , srv1589005003:84 , srv1725270595:84 , srv1830003154:84 , srv1863798956:84 , srv1875639243:84 , srv2033429542:84 , srv2033842659:84 , srv219053066:84 , srv255295496:84 , srv33713251:84 , srv356115035:84 , srv407904598:84 , srv511253286:84 , srv581120831:84 , srv612442452:84 , srv658287985:84 , srv732350700:84 , srv798647810:84 , srv80544750:84 , srv805945570:84 , srv898100240:84 , srv926456104:84 , srv1118894629:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-04-28 04:53:26,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-28 04:53:26,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1863798956=11, srv219053066=15, srv1118894629=1, srv1275809153=4, srv1358209742=5, srv658287985=23, srv255295496=16, srv407904598=19, srv114232896=2, srv898100240=28, srv1875639243=12, srv1117860974=0, srv1519099097=7, srv926456104=29, srv356115035=18, srv80544750=26, srv1376234994=6, srv732350700=24, srv1150614122=3, srv805945570=27, srv1830003154=10, srv581120831=21, srv612442452=22, srv2033429542=13, srv798647810=25, srv511253286=20, srv2033842659=14, srv33713251=17, srv1589005003=8, srv1725270595=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-28 04:53:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-28 04:53:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-28 04:53:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-28 04:53:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-28 04:53:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-28 04:53:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-28 04:53:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-28 04:53:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-28 04:53:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-28 04:53:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-28 04:53:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-28 04:53:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-28 04:53:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-28 04:53:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-28 04:53:26,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-28 04:53:26,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-28 04:53:26,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-28 04:53:26,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-28 04:53:26,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-28 04:53:26,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-28 04:53:26,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-28 04:53:26,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-28 04:53:26,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-28 04:53:26,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-28 04:53:26,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-28 04:53:26,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-28 04:53:26,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-28 04:53:26,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-28 04:53:26,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-28 04:53:26,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-28 04:53:26,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-28 04:53:26,253 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1706720990818273, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-28 04:53:26,253 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912730825475529, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1706720990818273, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-04-28 04:55:26,242 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9711343 different iterations.  Found a solution that moves 1848 regions; Going from a computed imbalance of 0.9912730825475529 to a new imbalance of 4.313324190526491E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6844444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-28 04:55:26,250 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1117860974:90 , srv1118894629:90 , srv114232896:90 , srv1150614122:90 , srv1275809153:90 , srv1358209742:90 , srv1376234994:90 , srv1519099097:90 , srv1589005003:90 , srv1725270595:90 , srv1830003154:90 , srv1863798956:90 , srv1875639243:90 , srv2033429542:90 , srv2033842659:90 , srv219053066:90 , srv255295496:90 , srv33713251:90 , srv356115035:90 , srv407904598:90 , srv511253286:90 , srv581120831:90 , srv612442452:90 , srv658287985:90 , srv732350700:90 , srv798647810:90 , srv80544750:90 , srv805945570:90 , srv898100240:90 , srv926456104:90 }
2023-04-28 04:55:26,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-28 04:55:26,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1863798956=11, srv219053066=15, srv1118894629=1, srv1275809153=4, srv1358209742=5, srv658287985=23, srv255295496=16, srv407904598=19, srv114232896=2, srv898100240=28, srv1875639243=12, srv1117860974=0, srv1519099097=7, srv926456104=29, srv356115035=18, srv80544750=26, srv1376234994=6, srv732350700=24, srv1150614122=3, srv805945570=27, srv1830003154=10, srv581120831=21, srv612442452=22, srv2033429542=13, srv798647810=25, srv511253286=20, srv2033842659=14, srv33713251=17, srv1589005003=8, srv1725270595=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-28 04:55:26,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-28 04:55:26,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-28 04:55:26,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-28 04:55:26,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-28 04:55:26,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-28 04:55:26,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-28 04:55:26,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-28 04:55:26,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-28 04:55:26,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-28 04:55:26,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-28 04:55:26,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-28 04:55:26,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-28 04:55:26,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-28 04:55:26,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-28 04:55:26,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-28 04:55:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-28 04:55:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-28 04:55:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-28 04:55:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-28 04:55:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-28 04:55:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-28 04:55:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-28 04:55:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-28 04:55:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-28 04:55:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-28 04:55:26,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-28 04:55:26,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-28 04:55:26,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-28 04:55:26,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-28 04:55:26,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-28 04:55:26,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-28 04:55:26,273 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-28 04:55:26,274 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1117860974:90 , srv1118894629:90 , srv114232896:90 , srv1150614122:90 , srv1275809153:90 , srv1358209742:90 , srv1376234994:90 , srv1519099097:90 , srv1589005003:90 , srv1725270595:90 , srv1830003154:90 , srv1863798956:90 , srv1875639243:90 , srv2033429542:90 , srv2033842659:90 , srv219053066:90 , srv255295496:90 , srv33713251:90 , srv356115035:90 , srv407904598:90 , srv511253286:90 , srv581120831:90 , srv612442452:90 , srv658287985:90 , srv732350700:90 , srv798647810:90 , srv80544750:90 , srv805945570:90 , srv898100240:90 , srv926456104:90 }
2023-04-28 04:55:26,287 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=188 (was 354), ProcessCount=171 (was 174), AvailableMemoryMB=10234 (was 11010)