2023-03-18 03:57:08,534 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-03-18 03:57:08,798 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-03-18 03:57:08,889 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-03-18 03:57:08,938 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-18 03:57:08,944 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-18 03:57:08,956 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=196, ProcessCount=167, AvailableMemoryMB=5633
2023-03-18 03:57:08,958 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-18 03:57:08,958 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-18 03:57:09,063 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv100651067:84 , srv1018301988:84 , srv1020778721:84 , srv1023327331:84 , srv1032037381:84 , srv1034901413:84 , srv1040469889:84 , srv1058474571:84 , srv1068051905:84 , srv1072411202:84 , srv1075061282:84 , srv1095818858:84 , srv1112608570:84 , srv1127704169:84 , srv1139151407:84 , srv1147128971:84 , srv1149507598:84 , srv1162035081:84 , srv1174944077:84 , srv1209257577:84 , srv1215307815:84 , srv1222363754:84 , srv1257445254:84 , srv1268110081:84 , srv1298932849:84 , srv1329735484:84 , srv1330767771:84 , srv1332127609:84 , srv1335429344:84 , srv1343347289:84 , srv1363150358:84 , srv1364576466:84 , srv1368178147:84 , srv1374089563:84 , srv1393603341:84 , srv1434865154:84 , srv1452782073:84 , srv1474436637:84 , srv1548153124:84 , srv1571880505:84 , srv1573302299:84 , srv160862716:84 , srv1620713480:84 , srv1669917143:84 , srv1677589871:84 , srv1699335104:84 , srv1756215805:84 , srv1761092312:84 , srv1779615324:84 , srv1783031294:84 , srv1805706937:84 , srv1869837819:84 , srv188018136:84 , srv1896987134:84 , srv192941972:84 , srv1966595973:84 , srv2001132728:84 , srv2030480651:84 , srv2037450657:84 , srv2042991460:84 , srv2048653160:84 , srv2090960340:84 , srv246786790:84 , srv250204132:84 , srv273964860:84 , srv306105175:84 , srv312933514:84 , srv348993066:84 , srv384026899:84 , srv391833793:84 , srv392310764:84 , srv41374052:84 , srv414434149:84 , srv415901370:84 , srv422856671:84 , srv456999326:84 , srv48572631:84 , srv495132867:84 , srv545164809:84 , srv551976529:84 , srv552269747:84 , srv562347178:84 , srv578048605:84 , srv578301552:84 , srv612857742:84 , srv69705237:84 , srv733533499:84 , srv739086212:84 , srv759959535:84 , srv762753534:84 , srv811454052:84 , srv828551727:84 , srv849608833:84 , srv874263724:84 , srv883794621:84 , srv913056234:84 , srv939386594:84 , srv973766710:84 , srv992419831:84 , srv1189707233:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-03-18 03:57:09,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-18 03:57:09,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv759959535=89, srv2048653160=61, srv1209257577=20, srv1329735484=26, srv1330767771=27, srv1149507598=16, srv1023327331=3, srv1020778721=2, srv939386594=97, srv1374089563=34, srv48572631=77, srv973766710=98, srv733533499=87, srv1332127609=28, srv246786790=63, srv1032037381=4, srv348993066=68, srv414434149=73, srv1966595973=56, srv828551727=92, srv811454052=91, srv1018301988=1, srv1343347289=30, srv1068051905=8, srv1699335104=46, srv1040469889=6, srv1215307815=21, srv1368178147=33, srv883794621=95, srv495132867=78, srv100651067=0, srv1761092312=48, srv188018136=53, srv849608833=93, srv1257445254=23, srv1669917143=44, srv992419831=99, srv1363150358=31, srv2042991460=60, srv1548153124=39, srv1112608570=12, srv415901370=74, srv762753534=90, srv1393603341=35, srv2001132728=57, srv1805706937=51, srv578048605=83, srv739086212=88, srv1139151407=14, srv1474436637=38, srv1783031294=50, srv1869837819=52, srv1174944077=18, srv1756215805=47, srv384026899=69, srv2090960340=62, srv1896987134=54, srv2037450657=59, srv160862716=42, srv422856671=75, srv551976529=80, srv1268110081=24, srv273964860=65, srv1222363754=22, srv1127704169=13, srv1620713480=43, srv1434865154=36, srv1571880505=40, srv1162035081=17, srv1677589871=45, srv312933514=67, srv41374052=72, srv69705237=86, srv2030480651=58, srv552269747=81, srv1452782073=37, srv612857742=85, srv1058474571=7, srv1189707233=19, srv1364576466=32, srv545164809=79, srv392310764=71, srv874263724=94, srv1075061282=10, srv1095818858=11, srv1147128971=15, srv1335429344=29, srv1779615324=49, srv1298932849=25, srv250204132=64, srv391833793=70, srv562347178=82, srv1573302299=41, srv456999326=76, srv1072411202=9, srv1034901413=5, srv578301552=84, srv913056234=96, srv192941972=55, srv306105175=66} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-18 03:57:09,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-18 03:57:09,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-18 03:57:09,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-18 03:57:09,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-18 03:57:09,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-18 03:57:09,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-18 03:57:09,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-18 03:57:09,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-18 03:57:09,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-18 03:57:09,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-18 03:57:09,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-18 03:57:09,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-18 03:57:09,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-18 03:57:09,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-18 03:57:09,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-18 03:57:09,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-18 03:57:09,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-18 03:57:09,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-18 03:57:09,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-18 03:57:09,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-18 03:57:09,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-18 03:57:09,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-18 03:57:09,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-18 03:57:09,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-18 03:57:09,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-18 03:57:09,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-18 03:57:09,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-18 03:57:09,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-18 03:57:09,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-18 03:57:09,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-18 03:57:09,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-18 03:57:09,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-18 03:57:09,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-18 03:57:09,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-18 03:57:09,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-18 03:57:09,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-18 03:57:09,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-18 03:57:09,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-18 03:57:09,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-18 03:57:09,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-18 03:57:09,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-18 03:57:09,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-18 03:57:09,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-18 03:57:09,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-18 03:57:09,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-18 03:57:09,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-18 03:57:09,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-18 03:57:09,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-18 03:57:09,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-18 03:57:09,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-18 03:57:09,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-18 03:57:09,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-18 03:57:09,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-18 03:57:09,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-18 03:57:09,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-18 03:57:09,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-18 03:57:09,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-18 03:57:09,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-18 03:57:09,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-18 03:57:09,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-18 03:57:09,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-18 03:57:09,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-18 03:57:09,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-18 03:57:09,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-18 03:57:09,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-18 03:57:09,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-18 03:57:09,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-18 03:57:09,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-18 03:57:09,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-18 03:57:09,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-18 03:57:09,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-18 03:57:09,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-18 03:57:09,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-18 03:57:09,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-18 03:57:09,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-18 03:57:09,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-18 03:57:09,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-18 03:57:09,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-18 03:57:09,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-18 03:57:09,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-18 03:57:09,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-18 03:57:09,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-18 03:57:09,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-18 03:57:09,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-18 03:57:09,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-18 03:57:09,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-18 03:57:09,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-18 03:57:09,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-18 03:57:09,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-18 03:57:09,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-18 03:57:09,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-18 03:57:09,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-18 03:57:09,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-18 03:57:09,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-18 03:57:09,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-18 03:57:09,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-18 03:57:09,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-18 03:57:09,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-18 03:57:09,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-18 03:57:09,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-18 03:57:09,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-18 03:57:09,363 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 03:57:09,363 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-18 03:57:09,363 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-18 03:57:09,742 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 564 ms to try 5000 different iterations.  Found a solution that moves 4704 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17621025562390316. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041623434361737176, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057292053105522545, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5226666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041623434361737176, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17116666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24016666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 03:57:09,750 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1330767771:81 , srv1368178147:81 , srv100651067:82 , srv1072411202:82 , srv2042991460:82 , srv874263724:82 , srv1032037381:83 , srv1374089563:83 , srv1677589871:83 , srv348993066:83 , srv456999326:83 , srv48572631:83 , srv552269747:83 , srv578048605:83 , srv1020778721:84 , srv1548153124:84 , srv160862716:84 , srv1761092312:84 , srv1779615324:84 , srv2090960340:84 , srv306105175:84 , srv69705237:84 , srv992419831:84 , srv1058474571:85 , srv1075061282:85 , srv1112608570:85 , srv1139151407:85 , srv1149507598:85 , srv1174944077:85 , srv1257445254:85 , srv1298932849:85 , srv1332127609:85 , srv1434865154:85 , srv1669917143:85 , srv1896987134:85 , srv1966595973:85 , srv2037450657:85 , srv578301552:85 , srv759959535:85 , srv913056234:85 , srv1018301988:86 , srv1023327331:86 , srv1034901413:86 , srv1127704169:86 , srv1162035081:86 , srv1364576466:86 , srv1393603341:86 , srv1620713480:86 , srv1756215805:86 , srv192941972:86 , srv273964860:86 , srv495132867:86 , srv551976529:86 , srv733533499:86 , srv939386594:86 , srv973766710:86 , srv1095818858:87 , srv1209257577:87 , srv1329735484:87 , srv1363150358:87 , srv1571880505:87 , srv1783031294:87 , srv1869837819:87 , srv2030480651:87 , srv250204132:87 , srv312933514:87 , srv391833793:87 , srv392310764:87 , srv811454052:87 , srv849608833:87 , srv883794621:87 , srv1068051905:88 , srv1215307815:88 , srv1335429344:88 , srv1452782073:88 , srv188018136:88 , srv2001132728:88 , srv2048653160:88 , srv246786790:88 , srv384026899:88 , srv739086212:88 , srv762753534:88 , srv1040469889:89 , srv1222363754:89 , srv1268110081:89 , srv414434149:89 , srv562347178:89 , srv612857742:89 , srv828551727:89 , srv41374052:90 , srv422856671:90 , srv545164809:90 , srv1474436637:91 , srv1573302299:91 , srv1699335104:91 , srv1343347289:92 , srv1805706937:92 , srv1147128971:93 , srv415901370:93 , srv1189707233:460 }
2023-03-18 03:57:09,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-18 03:57:09,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv759959535=89, srv2048653160=61, srv1209257577=20, srv1329735484=26, srv1330767771=27, srv1149507598=16, srv1023327331=3, srv1020778721=2, srv939386594=97, srv1374089563=34, srv48572631=77, srv973766710=98, srv733533499=87, srv1332127609=28, srv246786790=63, srv1032037381=4, srv348993066=68, srv414434149=73, srv1966595973=56, srv828551727=92, srv811454052=91, srv1018301988=1, srv1343347289=30, srv1068051905=8, srv1699335104=46, srv1040469889=6, srv1215307815=21, srv1368178147=33, srv883794621=95, srv495132867=78, srv100651067=0, srv1761092312=48, srv188018136=53, srv849608833=93, srv1257445254=23, srv1669917143=44, srv992419831=99, srv1363150358=31, srv2042991460=60, srv1548153124=39, srv1112608570=12, srv415901370=74, srv762753534=90, srv1393603341=35, srv2001132728=57, srv1805706937=51, srv578048605=83, srv739086212=88, srv1139151407=14, srv1474436637=38, srv1783031294=50, srv1869837819=52, srv1174944077=18, srv1756215805=47, srv384026899=69, srv2090960340=62, srv1896987134=54, srv2037450657=59, srv160862716=42, srv422856671=75, srv551976529=80, srv1268110081=24, srv273964860=65, srv1222363754=22, srv1127704169=13, srv1620713480=43, srv1434865154=36, srv1571880505=40, srv1162035081=17, srv1677589871=45, srv312933514=67, srv41374052=72, srv69705237=86, srv2030480651=58, srv552269747=81, srv1452782073=37, srv612857742=85, srv1058474571=7, srv1189707233=19, srv1364576466=32, srv545164809=79, srv392310764=71, srv874263724=94, srv1075061282=10, srv1095818858=11, srv1147128971=15, srv1335429344=29, srv1779615324=49, srv1298932849=25, srv250204132=64, srv391833793=70, srv562347178=82, srv1573302299=41, srv456999326=76, srv1072411202=9, srv1034901413=5, srv578301552=84, srv913056234=96, srv192941972=55, srv306105175=66} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-18 03:57:09,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-18 03:57:09,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-18 03:57:09,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-18 03:57:09,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-18 03:57:09,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-18 03:57:09,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-18 03:57:09,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-18 03:57:09,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-18 03:57:09,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-18 03:57:09,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-18 03:57:09,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-18 03:57:09,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-18 03:57:09,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-18 03:57:09,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-18 03:57:09,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-18 03:57:09,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-18 03:57:09,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-18 03:57:09,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-18 03:57:09,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-18 03:57:09,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-18 03:57:09,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-18 03:57:09,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-18 03:57:09,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-18 03:57:09,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-18 03:57:09,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-18 03:57:09,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-18 03:57:09,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-18 03:57:09,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-18 03:57:09,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-18 03:57:09,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-18 03:57:09,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-18 03:57:09,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-18 03:57:09,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-18 03:57:09,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-18 03:57:09,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-18 03:57:09,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-18 03:57:09,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-18 03:57:09,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-18 03:57:09,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-18 03:57:09,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-18 03:57:09,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-18 03:57:09,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-18 03:57:09,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-18 03:57:09,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-18 03:57:09,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-18 03:57:09,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-18 03:57:09,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-18 03:57:09,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-18 03:57:09,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-18 03:57:09,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-18 03:57:09,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-18 03:57:09,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-18 03:57:09,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-18 03:57:09,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-18 03:57:09,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-18 03:57:09,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-18 03:57:09,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-18 03:57:09,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-18 03:57:09,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-18 03:57:09,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-18 03:57:09,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-18 03:57:09,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-18 03:57:09,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-18 03:57:09,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-18 03:57:09,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-18 03:57:09,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-18 03:57:09,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-18 03:57:09,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-18 03:57:09,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-18 03:57:09,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-18 03:57:09,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-18 03:57:09,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-18 03:57:09,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-18 03:57:09,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-18 03:57:09,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-18 03:57:09,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-18 03:57:09,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-18 03:57:09,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-18 03:57:09,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-18 03:57:09,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-18 03:57:09,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-18 03:57:09,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-18 03:57:09,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-18 03:57:09,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-18 03:57:09,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-18 03:57:09,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-18 03:57:09,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-18 03:57:09,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-18 03:57:09,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-18 03:57:09,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-18 03:57:09,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-18 03:57:09,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-18 03:57:09,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-18 03:57:09,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-18 03:57:09,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-18 03:57:09,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-18 03:57:09,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-18 03:57:09,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-18 03:57:09,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-18 03:57:09,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-18 03:57:09,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-18 03:57:09,883 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041623434361737176, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057292053105522545, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041623434361737176, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17116666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24016666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 03:57:09,883 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-18 03:57:09,883 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1761773175119028, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041623434361737176, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.057292053105522545, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041623434361737176, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17116666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24016666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-18 03:57:10,072 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 287 ms to try 5000 different iterations.  Found a solution that moves 2626 regions; Going from a computed imbalance of 0.1761773175119028 to a new imbalance of 0.016488705200042884. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020145994141147615); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03239071239040984, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2917777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020145994141147615); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04608333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 03:57:10,081 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1147128971:78 , srv415901370:78 , srv1343347289:79 , srv1474436637:80 , srv1805706937:80 , srv422856671:80 , srv1222363754:81 , srv1573302299:81 , srv41374052:82 , srv1040469889:83 , srv1335429344:83 , srv414434149:83 , srv612857742:83 , srv828551727:83 , srv1068051905:84 , srv1209257577:84 , srv1268110081:84 , srv1329735484:84 , srv1452782073:84 , srv1783031294:84 , srv2030480651:84 , srv2048653160:84 , srv246786790:84 , srv250204132:84 , srv391833793:84 , srv392310764:84 , srv545164809:84 , srv733533499:84 , srv739086212:84 , srv973766710:84 , srv1034901413:85 , srv1095818858:85 , srv1139151407:85 , srv1215307815:85 , srv1699335104:85 , srv188018136:85 , srv562347178:85 , srv762753534:85 , srv1018301988:86 , srv1020778721:86 , srv1058474571:86 , srv1112608570:86 , srv1127704169:86 , srv1162035081:86 , srv1298932849:86 , srv1363150358:86 , srv1364576466:86 , srv1756215805:86 , srv1869837819:86 , srv192941972:86 , srv2037450657:86 , srv495132867:86 , srv578301552:86 , srv811454052:86 , srv849608833:86 , srv883794621:86 , srv913056234:86 , srv1174944077:87 , srv1257445254:87 , srv1434865154:87 , srv1548153124:87 , srv1779615324:87 , srv2001132728:87 , srv2090960340:87 , srv312933514:87 , srv551976529:87 , srv759959535:87 , srv939386594:87 , srv1023327331:88 , srv1075061282:88 , srv1149507598:88 , srv1571880505:88 , srv160862716:88 , srv1669917143:88 , srv1677589871:88 , srv1966595973:88 , srv273964860:88 , srv384026899:88 , srv992419831:88 , srv1072411202:89 , srv1330767771:89 , srv1332127609:89 , srv1393603341:89 , srv1896987134:89 , srv2042991460:89 , srv306105175:89 , srv348993066:89 , srv48572631:89 , srv552269747:89 , srv578048605:89 , srv69705237:89 , srv874263724:89 , srv100651067:90 , srv1032037381:90 , srv1374089563:90 , srv1620713480:90 , srv1761092312:90 , srv456999326:90 , srv1368178147:93 , srv1189707233:493 }
2023-03-18 03:57:10,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-18 03:57:10,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv759959535=89, srv2048653160=61, srv1209257577=20, srv1329735484=26, srv1330767771=27, srv1149507598=16, srv1023327331=3, srv1020778721=2, srv939386594=97, srv1374089563=34, srv48572631=77, srv973766710=98, srv733533499=87, srv1332127609=28, srv246786790=63, srv1032037381=4, srv348993066=68, srv414434149=73, srv1966595973=56, srv828551727=92, srv811454052=91, srv1018301988=1, srv1343347289=30, srv1068051905=8, srv1699335104=46, srv1040469889=6, srv1215307815=21, srv1368178147=33, srv883794621=95, srv495132867=78, srv100651067=0, srv1761092312=48, srv188018136=53, srv849608833=93, srv1257445254=23, srv1669917143=44, srv992419831=99, srv1363150358=31, srv2042991460=60, srv1548153124=39, srv1112608570=12, srv415901370=74, srv762753534=90, srv1393603341=35, srv2001132728=57, srv1805706937=51, srv578048605=83, srv739086212=88, srv1139151407=14, srv1474436637=38, srv1783031294=50, srv1869837819=52, srv1174944077=18, srv1756215805=47, srv384026899=69, srv2090960340=62, srv1896987134=54, srv2037450657=59, srv160862716=42, srv422856671=75, srv551976529=80, srv1268110081=24, srv273964860=65, srv1222363754=22, srv1127704169=13, srv1620713480=43, srv1434865154=36, srv1571880505=40, srv1162035081=17, srv1677589871=45, srv312933514=67, srv41374052=72, srv69705237=86, srv2030480651=58, srv552269747=81, srv1452782073=37, srv612857742=85, srv1058474571=7, srv1189707233=19, srv1364576466=32, srv545164809=79, srv392310764=71, srv874263724=94, srv1075061282=10, srv1095818858=11, srv1147128971=15, srv1335429344=29, srv1779615324=49, srv1298932849=25, srv250204132=64, srv391833793=70, srv562347178=82, srv1573302299=41, srv456999326=76, srv1072411202=9, srv1034901413=5, srv578301552=84, srv913056234=96, srv192941972=55, srv306105175=66} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-18 03:57:10,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-18 03:57:10,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-18 03:57:10,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-18 03:57:10,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-18 03:57:10,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-18 03:57:10,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-18 03:57:10,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-18 03:57:10,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-18 03:57:10,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-18 03:57:10,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-18 03:57:10,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-18 03:57:10,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-18 03:57:10,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-18 03:57:10,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-18 03:57:10,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-18 03:57:10,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-18 03:57:10,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-18 03:57:10,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-18 03:57:10,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-18 03:57:10,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-18 03:57:10,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-18 03:57:10,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-18 03:57:10,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-18 03:57:10,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-18 03:57:10,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-18 03:57:10,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-18 03:57:10,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-18 03:57:10,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-18 03:57:10,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-18 03:57:10,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-18 03:57:10,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-18 03:57:10,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-18 03:57:10,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-18 03:57:10,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-18 03:57:10,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-18 03:57:10,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-18 03:57:10,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-18 03:57:10,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-18 03:57:10,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-18 03:57:10,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-18 03:57:10,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-18 03:57:10,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-18 03:57:10,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-18 03:57:10,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-18 03:57:10,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-18 03:57:10,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-18 03:57:10,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-18 03:57:10,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-18 03:57:10,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-18 03:57:10,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-18 03:57:10,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-18 03:57:10,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-18 03:57:10,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-18 03:57:10,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-18 03:57:10,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-18 03:57:10,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-18 03:57:10,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-18 03:57:10,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-18 03:57:10,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-18 03:57:10,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-18 03:57:10,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-18 03:57:10,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-18 03:57:10,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-18 03:57:10,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-18 03:57:10,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-18 03:57:10,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-18 03:57:10,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-18 03:57:10,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-18 03:57:10,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-18 03:57:10,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-18 03:57:10,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-18 03:57:10,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-18 03:57:10,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-18 03:57:10,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-18 03:57:10,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-18 03:57:10,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-18 03:57:10,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-18 03:57:10,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-18 03:57:10,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-18 03:57:10,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-18 03:57:10,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-18 03:57:10,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-18 03:57:10,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-18 03:57:10,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-18 03:57:10,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-18 03:57:10,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-18 03:57:10,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-18 03:57:10,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-18 03:57:10,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-18 03:57:10,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-18 03:57:10,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-18 03:57:10,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-18 03:57:10,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-18 03:57:10,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-18 03:57:10,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-18 03:57:10,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-18 03:57:10,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-18 03:57:10,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-18 03:57:10,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-18 03:57:10,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-18 03:57:10,106 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-18 03:57:10,195 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020145994141147615); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03239071239040984, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020145994141147615); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04608333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 03:57:10,195 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-18 03:57:10,195 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016470317555035866, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020145994141147615); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03239071239040984, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020145994141147615); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04608333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-18 03:57:10,366 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 250 ms to try 5000 different iterations.  Found a solution that moves 1036 regions; Going from a computed imbalance of 0.016470317555035866 to a new imbalance of 0.0010589794024205073. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005213965342504877); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009732678359956285); MoveCostFunction : (multiplier=7.0, imbalance=0.11511111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005213965342504877); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 03:57:10,368 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1620713480:81 , srv1023327331:82 , srv1393603341:82 , srv1699335104:82 , srv384026899:82 , srv1368178147:83 , srv1571880505:83 , srv1761092312:83 , srv1896987134:83 , srv2001132728:83 , srv849608833:83 , srv1032037381:84 , srv1149507598:84 , srv1162035081:84 , srv1268110081:84 , srv1332127609:84 , srv1374089563:84 , srv1669917143:84 , srv1966595973:84 , srv273964860:84 , srv312933514:84 , srv456999326:84 , srv545164809:84 , srv562347178:84 , srv811454052:84 , srv883794621:84 , srv1068051905:85 , srv1075061282:85 , srv1174944077:85 , srv1215307815:85 , srv1257445254:85 , srv1434865154:85 , srv160862716:85 , srv1805706937:85 , srv1869837819:85 , srv188018136:85 , srv192941972:85 , srv2048653160:85 , srv306105175:85 , srv414434149:85 , srv495132867:85 , srv612857742:85 , srv69705237:85 , srv762753534:85 , srv828551727:85 , srv939386594:85 , srv992419831:85 , srv100651067:86 , srv1018301988:86 , srv1034901413:86 , srv1040469889:86 , srv1095818858:86 , srv1112608570:86 , srv1329735484:86 , srv1343347289:86 , srv1363150358:86 , srv1452782073:86 , srv1474436637:86 , srv1548153124:86 , srv1573302299:86 , srv1756215805:86 , srv1779615324:86 , srv1783031294:86 , srv2037450657:86 , srv2090960340:86 , srv246786790:86 , srv250204132:86 , srv348993066:86 , srv392310764:86 , srv48572631:86 , srv551976529:86 , srv552269747:86 , srv578048605:86 , srv578301552:86 , srv739086212:86 , srv759959535:86 , srv913056234:86 , srv1020778721:87 , srv1058474571:87 , srv1072411202:87 , srv1127704169:87 , srv1139151407:87 , srv1147128971:87 , srv1209257577:87 , srv1298932849:87 , srv1330767771:87 , srv1335429344:87 , srv1364576466:87 , srv1677589871:87 , srv2030480651:87 , srv2042991460:87 , srv41374052:87 , srv415901370:87 , srv422856671:87 , srv874263724:87 , srv973766710:87 , srv1222363754:88 , srv391833793:88 , srv733533499:88 , srv1189707233:551 }
2023-03-18 03:57:10,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-18 03:57:10,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv759959535=89, srv2048653160=61, srv1209257577=20, srv1329735484=26, srv1330767771=27, srv1149507598=16, srv1023327331=3, srv1020778721=2, srv939386594=97, srv1374089563=34, srv48572631=77, srv973766710=98, srv733533499=87, srv1332127609=28, srv246786790=63, srv1032037381=4, srv348993066=68, srv414434149=73, srv1966595973=56, srv828551727=92, srv811454052=91, srv1018301988=1, srv1343347289=30, srv1068051905=8, srv1699335104=46, srv1040469889=6, srv1215307815=21, srv1368178147=33, srv883794621=95, srv495132867=78, srv100651067=0, srv1761092312=48, srv188018136=53, srv849608833=93, srv1257445254=23, srv1669917143=44, srv992419831=99, srv1363150358=31, srv2042991460=60, srv1548153124=39, srv1112608570=12, srv415901370=74, srv762753534=90, srv1393603341=35, srv2001132728=57, srv1805706937=51, srv578048605=83, srv739086212=88, srv1139151407=14, srv1474436637=38, srv1783031294=50, srv1869837819=52, srv1174944077=18, srv1756215805=47, srv384026899=69, srv2090960340=62, srv1896987134=54, srv2037450657=59, srv160862716=42, srv422856671=75, srv551976529=80, srv1268110081=24, srv273964860=65, srv1222363754=22, srv1127704169=13, srv1620713480=43, srv1434865154=36, srv1571880505=40, srv1162035081=17, srv1677589871=45, srv312933514=67, srv41374052=72, srv69705237=86, srv2030480651=58, srv552269747=81, srv1452782073=37, srv612857742=85, srv1058474571=7, srv1189707233=19, srv1364576466=32, srv545164809=79, srv392310764=71, srv874263724=94, srv1075061282=10, srv1095818858=11, srv1147128971=15, srv1335429344=29, srv1779615324=49, srv1298932849=25, srv250204132=64, srv391833793=70, srv562347178=82, srv1573302299=41, srv456999326=76, srv1072411202=9, srv1034901413=5, srv578301552=84, srv913056234=96, srv192941972=55, srv306105175=66} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-18 03:57:10,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-18 03:57:10,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-18 03:57:10,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-18 03:57:10,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-18 03:57:10,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-18 03:57:10,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-18 03:57:10,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-18 03:57:10,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-18 03:57:10,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-18 03:57:10,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-18 03:57:10,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-18 03:57:10,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-18 03:57:10,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-18 03:57:10,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-18 03:57:10,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-18 03:57:10,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-18 03:57:10,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-18 03:57:10,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-18 03:57:10,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-18 03:57:10,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-18 03:57:10,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-18 03:57:10,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-18 03:57:10,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-18 03:57:10,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-18 03:57:10,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-18 03:57:10,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-18 03:57:10,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-18 03:57:10,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-18 03:57:10,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-18 03:57:10,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-18 03:57:10,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-18 03:57:10,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-18 03:57:10,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-18 03:57:10,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-18 03:57:10,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-18 03:57:10,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-18 03:57:10,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-18 03:57:10,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-18 03:57:10,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-18 03:57:10,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-18 03:57:10,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-18 03:57:10,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-18 03:57:10,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-18 03:57:10,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-18 03:57:10,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-18 03:57:10,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-18 03:57:10,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-18 03:57:10,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-18 03:57:10,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-18 03:57:10,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-18 03:57:10,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-18 03:57:10,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-18 03:57:10,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-18 03:57:10,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-18 03:57:10,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-18 03:57:10,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-18 03:57:10,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-18 03:57:10,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-18 03:57:10,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-18 03:57:10,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-18 03:57:10,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-18 03:57:10,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-18 03:57:10,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-18 03:57:10,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-18 03:57:10,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-18 03:57:10,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-18 03:57:10,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-18 03:57:10,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-18 03:57:10,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-18 03:57:10,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-18 03:57:10,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-18 03:57:10,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-18 03:57:10,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-18 03:57:10,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-18 03:57:10,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-18 03:57:10,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-18 03:57:10,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-18 03:57:10,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-18 03:57:10,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-18 03:57:10,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-18 03:57:10,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-18 03:57:10,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-18 03:57:10,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-18 03:57:10,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-18 03:57:10,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-18 03:57:10,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-18 03:57:10,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-18 03:57:10,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-18 03:57:10,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-18 03:57:10,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-18 03:57:10,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-18 03:57:10,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-18 03:57:10,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-18 03:57:10,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-18 03:57:10,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-18 03:57:10,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-18 03:57:10,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-18 03:57:10,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-18 03:57:10,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-18 03:57:10,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-18 03:57:10,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-18 03:57:10,472 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005213965342504877); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009732678359956285); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005213965342504877); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 03:57:10,472 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-18 03:57:10,472 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0010517251753728036, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005213965342504877); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009732678359956285); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005213965342504877); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-18 03:57:10,694 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 297 ms to try 5000 different iterations.  Found a solution that moves 252 regions; Going from a computed imbalance of 0.0010517251753728036 to a new imbalance of 4.98177699588301E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.028, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 03:57:10,695 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1127704169:83 , srv1147128971:83 , srv1363150358:83 , srv1364576466:83 , srv391833793:83 , srv41374052:83 , srv551976529:83 , srv100651067:84 , srv1040469889:84 , srv1058474571:84 , srv1072411202:84 , srv1075061282:84 , srv1209257577:84 , srv1215307815:84 , srv1222363754:84 , srv1298932849:84 , srv1329735484:84 , srv1332127609:84 , srv1335429344:84 , srv1393603341:84 , srv1548153124:84 , srv1573302299:84 , srv1677589871:84 , srv1756215805:84 , srv1869837819:84 , srv188018136:84 , srv2001132728:84 , srv2042991460:84 , srv246786790:84 , srv273964860:84 , srv306105175:84 , srv312933514:84 , srv348993066:84 , srv384026899:84 , srv415901370:84 , srv48572631:84 , srv545164809:84 , srv552269747:84 , srv562347178:84 , srv578048605:84 , srv578301552:84 , srv69705237:84 , srv733533499:84 , srv739086212:84 , srv759959535:84 , srv762753534:84 , srv811454052:84 , srv874263724:84 , srv913056234:84 , srv939386594:84 , srv992419831:84 , srv1018301988:85 , srv1020778721:85 , srv1023327331:85 , srv1032037381:85 , srv1034901413:85 , srv1095818858:85 , srv1112608570:85 , srv1139151407:85 , srv1149507598:85 , srv1174944077:85 , srv1257445254:85 , srv1268110081:85 , srv1330767771:85 , srv1343347289:85 , srv1368178147:85 , srv1374089563:85 , srv1434865154:85 , srv1452782073:85 , srv1474436637:85 , srv1571880505:85 , srv160862716:85 , srv1620713480:85 , srv1669917143:85 , srv1699335104:85 , srv1761092312:85 , srv1779615324:85 , srv1783031294:85 , srv1805706937:85 , srv1896987134:85 , srv1966595973:85 , srv2030480651:85 , srv2037450657:85 , srv2048653160:85 , srv2090960340:85 , srv250204132:85 , srv392310764:85 , srv414434149:85 , srv422856671:85 , srv456999326:85 , srv495132867:85 , srv828551727:85 , srv883794621:85 , srv973766710:85 , srv1068051905:86 , srv1162035081:86 , srv192941972:86 , srv612857742:86 , srv849608833:86 , srv1189707233:638 }
2023-03-18 03:57:10,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-18 03:57:10,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv759959535=89, srv2048653160=61, srv1209257577=20, srv1329735484=26, srv1330767771=27, srv1149507598=16, srv1023327331=3, srv1020778721=2, srv939386594=97, srv1374089563=34, srv48572631=77, srv973766710=98, srv733533499=87, srv1332127609=28, srv246786790=63, srv1032037381=4, srv348993066=68, srv414434149=73, srv1966595973=56, srv828551727=92, srv811454052=91, srv1018301988=1, srv1343347289=30, srv1068051905=8, srv1699335104=46, srv1040469889=6, srv1215307815=21, srv1368178147=33, srv883794621=95, srv495132867=78, srv100651067=0, srv1761092312=48, srv188018136=53, srv849608833=93, srv1257445254=23, srv1669917143=44, srv992419831=99, srv1363150358=31, srv2042991460=60, srv1548153124=39, srv1112608570=12, srv415901370=74, srv762753534=90, srv1393603341=35, srv2001132728=57, srv1805706937=51, srv578048605=83, srv739086212=88, srv1139151407=14, srv1474436637=38, srv1783031294=50, srv1869837819=52, srv1174944077=18, srv1756215805=47, srv384026899=69, srv2090960340=62, srv1896987134=54, srv2037450657=59, srv160862716=42, srv422856671=75, srv551976529=80, srv1268110081=24, srv273964860=65, srv1222363754=22, srv1127704169=13, srv1620713480=43, srv1434865154=36, srv1571880505=40, srv1162035081=17, srv1677589871=45, srv312933514=67, srv41374052=72, srv69705237=86, srv2030480651=58, srv552269747=81, srv1452782073=37, srv612857742=85, srv1058474571=7, srv1189707233=19, srv1364576466=32, srv545164809=79, srv392310764=71, srv874263724=94, srv1075061282=10, srv1095818858=11, srv1147128971=15, srv1335429344=29, srv1779615324=49, srv1298932849=25, srv250204132=64, srv391833793=70, srv562347178=82, srv1573302299=41, srv456999326=76, srv1072411202=9, srv1034901413=5, srv578301552=84, srv913056234=96, srv192941972=55, srv306105175=66} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-18 03:57:10,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-18 03:57:10,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-18 03:57:10,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-18 03:57:10,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-18 03:57:10,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-18 03:57:10,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-18 03:57:10,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-18 03:57:10,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-18 03:57:10,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-18 03:57:10,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-18 03:57:10,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-18 03:57:10,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-18 03:57:10,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-18 03:57:10,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-18 03:57:10,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-18 03:57:10,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-18 03:57:10,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-18 03:57:10,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-18 03:57:10,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-18 03:57:10,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-18 03:57:10,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-18 03:57:10,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-18 03:57:10,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-18 03:57:10,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-18 03:57:10,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-18 03:57:10,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-18 03:57:10,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-18 03:57:10,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-18 03:57:10,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-18 03:57:10,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-18 03:57:10,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-18 03:57:10,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-18 03:57:10,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-18 03:57:10,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-18 03:57:10,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-18 03:57:10,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-18 03:57:10,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-18 03:57:10,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-18 03:57:10,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-18 03:57:10,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-18 03:57:10,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-18 03:57:10,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-18 03:57:10,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-18 03:57:10,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-18 03:57:10,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-18 03:57:10,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-18 03:57:10,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-18 03:57:10,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-18 03:57:10,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-18 03:57:10,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-18 03:57:10,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-18 03:57:10,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-18 03:57:10,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-18 03:57:10,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-18 03:57:10,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-18 03:57:10,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-18 03:57:10,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-18 03:57:10,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-18 03:57:10,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-18 03:57:10,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-18 03:57:10,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-18 03:57:10,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-18 03:57:10,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-18 03:57:10,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-18 03:57:10,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-18 03:57:10,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-18 03:57:10,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-18 03:57:10,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-18 03:57:10,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-18 03:57:10,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-18 03:57:10,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-18 03:57:10,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-18 03:57:10,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-18 03:57:10,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-18 03:57:10,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-18 03:57:10,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-18 03:57:10,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-18 03:57:10,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-18 03:57:10,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-18 03:57:10,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-18 03:57:10,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-18 03:57:10,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-18 03:57:10,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-18 03:57:10,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-18 03:57:10,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-18 03:57:10,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-18 03:57:10,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-18 03:57:10,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-18 03:57:10,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-18 03:57:10,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-18 03:57:10,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-18 03:57:10,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-18 03:57:10,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-18 03:57:10,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-18 03:57:10,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-18 03:57:10,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-18 03:57:10,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-18 03:57:10,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-18 03:57:10,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-18 03:57:10,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-18 03:57:10,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-18 03:57:10,796 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 03:57:10,796 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-18 03:57:10,797 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.805322824452381E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-18 03:57:10,948 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 227 ms to try 5000 different iterations.  Found a solution that moves 23 regions; Going from a computed imbalance of 4.805322824452381E-5 to a new imbalance of 8.420813428170152E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0025555555555555557); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 03:57:10,948 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1018301988:83 , srv1068051905:83 , srv1095818858:83 , srv1452782073:83 , srv1571880505:83 , srv1699335104:83 , srv192941972:83 , srv2030480651:83 , srv612857742:83 , srv100651067:84 , srv1020778721:84 , srv1032037381:84 , srv1034901413:84 , srv1040469889:84 , srv1058474571:84 , srv1072411202:84 , srv1075061282:84 , srv1112608570:84 , srv1127704169:84 , srv1139151407:84 , srv1149507598:84 , srv1162035081:84 , srv1174944077:84 , srv1209257577:84 , srv1215307815:84 , srv1222363754:84 , srv1257445254:84 , srv1268110081:84 , srv1298932849:84 , srv1330767771:84 , srv1332127609:84 , srv1335429344:84 , srv1343347289:84 , srv1363150358:84 , srv1364576466:84 , srv1368178147:84 , srv1374089563:84 , srv1434865154:84 , srv1474436637:84 , srv1573302299:84 , srv160862716:84 , srv1620713480:84 , srv1669917143:84 , srv1677589871:84 , srv1756215805:84 , srv1761092312:84 , srv1779615324:84 , srv1783031294:84 , srv1805706937:84 , srv1869837819:84 , srv188018136:84 , srv1896987134:84 , srv1966595973:84 , srv2001132728:84 , srv2037450657:84 , srv2042991460:84 , srv2048653160:84 , srv2090960340:84 , srv246786790:84 , srv250204132:84 , srv273964860:84 , srv306105175:84 , srv312933514:84 , srv348993066:84 , srv384026899:84 , srv391833793:84 , srv392310764:84 , srv41374052:84 , srv414434149:84 , srv415901370:84 , srv422856671:84 , srv456999326:84 , srv48572631:84 , srv495132867:84 , srv545164809:84 , srv551976529:84 , srv552269747:84 , srv562347178:84 , srv578048605:84 , srv69705237:84 , srv733533499:84 , srv739086212:84 , srv759959535:84 , srv762753534:84 , srv828551727:84 , srv849608833:84 , srv874263724:84 , srv883794621:84 , srv939386594:84 , srv973766710:84 , srv1023327331:85 , srv1147128971:85 , srv1329735484:85 , srv1393603341:85 , srv1548153124:85 , srv578301552:85 , srv811454052:85 , srv913056234:85 , srv992419831:85 , srv1189707233:684 }
2023-03-18 03:57:10,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-18 03:57:10,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv759959535=89, srv2048653160=61, srv1209257577=20, srv1329735484=26, srv1330767771=27, srv1149507598=16, srv1023327331=3, srv1020778721=2, srv939386594=97, srv1374089563=34, srv48572631=77, srv973766710=98, srv733533499=87, srv1332127609=28, srv246786790=63, srv1032037381=4, srv348993066=68, srv414434149=73, srv1966595973=56, srv828551727=92, srv811454052=91, srv1018301988=1, srv1343347289=30, srv1068051905=8, srv1699335104=46, srv1040469889=6, srv1215307815=21, srv1368178147=33, srv883794621=95, srv495132867=78, srv100651067=0, srv1761092312=48, srv188018136=53, srv849608833=93, srv1257445254=23, srv1669917143=44, srv992419831=99, srv1363150358=31, srv2042991460=60, srv1548153124=39, srv1112608570=12, srv415901370=74, srv762753534=90, srv1393603341=35, srv2001132728=57, srv1805706937=51, srv578048605=83, srv739086212=88, srv1139151407=14, srv1474436637=38, srv1783031294=50, srv1869837819=52, srv1174944077=18, srv1756215805=47, srv384026899=69, srv2090960340=62, srv1896987134=54, srv2037450657=59, srv160862716=42, srv422856671=75, srv551976529=80, srv1268110081=24, srv273964860=65, srv1222363754=22, srv1127704169=13, srv1620713480=43, srv1434865154=36, srv1571880505=40, srv1162035081=17, srv1677589871=45, srv312933514=67, srv41374052=72, srv69705237=86, srv2030480651=58, srv552269747=81, srv1452782073=37, srv612857742=85, srv1058474571=7, srv1189707233=19, srv1364576466=32, srv545164809=79, srv392310764=71, srv874263724=94, srv1075061282=10, srv1095818858=11, srv1147128971=15, srv1335429344=29, srv1779615324=49, srv1298932849=25, srv250204132=64, srv391833793=70, srv562347178=82, srv1573302299=41, srv456999326=76, srv1072411202=9, srv1034901413=5, srv578301552=84, srv913056234=96, srv192941972=55, srv306105175=66} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-18 03:57:10,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-18 03:57:10,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-18 03:57:10,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-18 03:57:10,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-18 03:57:10,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-18 03:57:10,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-18 03:57:10,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-18 03:57:10,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-18 03:57:10,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-18 03:57:10,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-18 03:57:10,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-18 03:57:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-18 03:57:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-18 03:57:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-18 03:57:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-18 03:57:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-18 03:57:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-18 03:57:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-18 03:57:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-18 03:57:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-18 03:57:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-18 03:57:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-18 03:57:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-18 03:57:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-18 03:57:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-18 03:57:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-18 03:57:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-18 03:57:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-18 03:57:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-18 03:57:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-18 03:57:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-18 03:57:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-18 03:57:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-18 03:57:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-18 03:57:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-18 03:57:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-18 03:57:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-18 03:57:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-18 03:57:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-18 03:57:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-18 03:57:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-18 03:57:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-18 03:57:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-18 03:57:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-18 03:57:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-18 03:57:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-18 03:57:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-18 03:57:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-18 03:57:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-18 03:57:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-18 03:57:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-18 03:57:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-18 03:57:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-18 03:57:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-18 03:57:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-18 03:57:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-18 03:57:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-18 03:57:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-18 03:57:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-18 03:57:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-18 03:57:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-18 03:57:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-18 03:57:10,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-18 03:57:10,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-18 03:57:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-18 03:57:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-18 03:57:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-18 03:57:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-18 03:57:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-18 03:57:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-18 03:57:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-18 03:57:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-18 03:57:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-18 03:57:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-18 03:57:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-18 03:57:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-18 03:57:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-18 03:57:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-18 03:57:10,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-18 03:57:11,050 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=8.259763985991403E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 03:57:11,050 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1018301988:83 , srv1068051905:83 , srv1095818858:83 , srv1452782073:83 , srv1571880505:83 , srv1699335104:83 , srv192941972:83 , srv2030480651:83 , srv612857742:83 , srv100651067:84 , srv1020778721:84 , srv1032037381:84 , srv1034901413:84 , srv1040469889:84 , srv1058474571:84 , srv1072411202:84 , srv1075061282:84 , srv1112608570:84 , srv1127704169:84 , srv1139151407:84 , srv1149507598:84 , srv1162035081:84 , srv1174944077:84 , srv1209257577:84 , srv1215307815:84 , srv1222363754:84 , srv1257445254:84 , srv1268110081:84 , srv1298932849:84 , srv1330767771:84 , srv1332127609:84 , srv1335429344:84 , srv1343347289:84 , srv1363150358:84 , srv1364576466:84 , srv1368178147:84 , srv1374089563:84 , srv1434865154:84 , srv1474436637:84 , srv1573302299:84 , srv160862716:84 , srv1620713480:84 , srv1669917143:84 , srv1677589871:84 , srv1756215805:84 , srv1761092312:84 , srv1779615324:84 , srv1783031294:84 , srv1805706937:84 , srv1869837819:84 , srv188018136:84 , srv1896987134:84 , srv1966595973:84 , srv2001132728:84 , srv2037450657:84 , srv2042991460:84 , srv2048653160:84 , srv2090960340:84 , srv246786790:84 , srv250204132:84 , srv273964860:84 , srv306105175:84 , srv312933514:84 , srv348993066:84 , srv384026899:84 , srv391833793:84 , srv392310764:84 , srv41374052:84 , srv414434149:84 , srv415901370:84 , srv422856671:84 , srv456999326:84 , srv48572631:84 , srv495132867:84 , srv545164809:84 , srv551976529:84 , srv552269747:84 , srv562347178:84 , srv578048605:84 , srv69705237:84 , srv733533499:84 , srv739086212:84 , srv759959535:84 , srv762753534:84 , srv828551727:84 , srv849608833:84 , srv874263724:84 , srv883794621:84 , srv939386594:84 , srv973766710:84 , srv1023327331:85 , srv1147128971:85 , srv1329735484:85 , srv1393603341:85 , srv1548153124:85 , srv578301552:85 , srv811454052:85 , srv913056234:85 , srv992419831:85 , srv1189707233:684 }
2023-03-18 03:57:11,073 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=196 (was 196), ProcessCount=167 (was 167), AvailableMemoryMB=5142 (was 5633)
2023-03-18 03:57:11,080 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=196, ProcessCount=167, AvailableMemoryMB=5142
2023-03-18 03:57:11,081 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-18 03:57:11,081 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-03-18 03:57:11,091 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1035892744:84 , srv115118322:84 , srv1182590348:84 , srv1218148296:84 , srv1512262731:84 , srv1546913940:84 , srv1680191977:84 , srv1728662114:84 , srv1784836577:84 , srv1787501753:84 , srv17934841:84 , srv1795663815:84 , srv1820444439:84 , srv1905615:84 , srv1951615401:84 , srv1999267071:84 , srv2053642107:84 , srv2100594539:84 , srv2119302600:84 , srv254385512:84 , srv432963357:84 , srv495396023:84 , srv56762210:84 , srv614338854:84 , srv74956759:84 , srv802945826:84 , srv867630448:84 , srv877009610:84 , srv95098899:84 , srv1132154943:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-03-18 03:57:11,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-18 03:57:11,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv56762210=23, srv802945826=26, srv877009610=28, srv2119302600=19, srv1999267071=16, srv17934841=11, srv1182590348=3, srv1784836577=9, srv95098899=29, srv254385512=20, srv432963357=21, srv1951615401=15, srv115118322=2, srv867630448=27, srv1132154943=1, srv1218148296=4, srv1546913940=6, srv1680191977=7, srv1905615=14, srv2100594539=18, srv1795663815=12, srv1035892744=0, srv1512262731=5, srv495396023=22, srv1787501753=10, srv2053642107=17, srv1728662114=8, srv1820444439=13, srv614338854=24, srv74956759=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-18 03:57:11,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-18 03:57:11,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-18 03:57:11,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-18 03:57:11,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-18 03:57:11,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-18 03:57:11,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-18 03:57:11,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-18 03:57:11,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-18 03:57:11,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-18 03:57:11,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-18 03:57:11,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-18 03:57:11,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-18 03:57:11,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-18 03:57:11,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-18 03:57:11,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-18 03:57:11,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-18 03:57:11,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-18 03:57:11,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-18 03:57:11,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-18 03:57:11,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-18 03:57:11,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-18 03:57:11,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-18 03:57:11,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-18 03:57:11,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-18 03:57:11,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-18 03:57:11,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-18 03:57:11,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-18 03:57:11,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-18 03:57:11,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-18 03:57:11,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-18 03:57:11,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-18 03:57:11,133 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1705842478979038, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 03:57:11,134 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912730548659316, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1705842478979038, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-03-18 03:59:11,113 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9455040 different iterations.  Found a solution that moves 1836 regions; Going from a computed imbalance of 0.9912730548659316 to a new imbalance of 4.2853155918867096E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.68, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 03:59:11,119 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1035892744:90 , srv1132154943:90 , srv115118322:90 , srv1182590348:90 , srv1218148296:90 , srv1512262731:90 , srv1546913940:90 , srv1680191977:90 , srv1728662114:90 , srv1784836577:90 , srv1787501753:90 , srv17934841:90 , srv1795663815:90 , srv1820444439:90 , srv1905615:90 , srv1951615401:90 , srv1999267071:90 , srv2053642107:90 , srv2100594539:90 , srv2119302600:90 , srv254385512:90 , srv432963357:90 , srv495396023:90 , srv56762210:90 , srv614338854:90 , srv74956759:90 , srv802945826:90 , srv867630448:90 , srv877009610:90 , srv95098899:90 }
2023-03-18 03:59:11,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-18 03:59:11,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv56762210=23, srv802945826=26, srv877009610=28, srv2119302600=19, srv1999267071=16, srv17934841=11, srv1182590348=3, srv1784836577=9, srv95098899=29, srv254385512=20, srv432963357=21, srv1951615401=15, srv115118322=2, srv867630448=27, srv1132154943=1, srv1218148296=4, srv1546913940=6, srv1680191977=7, srv1905615=14, srv2100594539=18, srv1795663815=12, srv1035892744=0, srv1512262731=5, srv495396023=22, srv1787501753=10, srv2053642107=17, srv1728662114=8, srv1820444439=13, srv614338854=24, srv74956759=25} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-18 03:59:11,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-18 03:59:11,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-18 03:59:11,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-18 03:59:11,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-18 03:59:11,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-18 03:59:11,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-18 03:59:11,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-18 03:59:11,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-18 03:59:11,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-18 03:59:11,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-18 03:59:11,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-18 03:59:11,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-18 03:59:11,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-18 03:59:11,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-18 03:59:11,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-18 03:59:11,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-18 03:59:11,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-18 03:59:11,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-18 03:59:11,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-18 03:59:11,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-18 03:59:11,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-18 03:59:11,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-18 03:59:11,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-18 03:59:11,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-18 03:59:11,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-18 03:59:11,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-18 03:59:11,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-18 03:59:11,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-18 03:59:11,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-18 03:59:11,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-18 03:59:11,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-18 03:59:11,136 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-18 03:59:11,137 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1035892744:90 , srv1132154943:90 , srv115118322:90 , srv1182590348:90 , srv1218148296:90 , srv1512262731:90 , srv1546913940:90 , srv1680191977:90 , srv1728662114:90 , srv1784836577:90 , srv1787501753:90 , srv17934841:90 , srv1795663815:90 , srv1820444439:90 , srv1905615:90 , srv1951615401:90 , srv1999267071:90 , srv2053642107:90 , srv2100594539:90 , srv2119302600:90 , srv254385512:90 , srv432963357:90 , srv495396023:90 , srv56762210:90 , srv614338854:90 , srv74956759:90 , srv802945826:90 , srv867630448:90 , srv877009610:90 , srv95098899:90 }
2023-03-18 03:59:11,147 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=190 (was 196), ProcessCount=164 (was 167), AvailableMemoryMB=5978 (was 5142) - AvailableMemoryMB LEAK? -