2023-04-18 12:55:43,544 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-04-18 12:55:43,955 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-04-18 12:55:44,091 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-04-18 12:55:44,169 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-18 12:55:44,181 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-18 12:55:44,208 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=386, ProcessCount=169, AvailableMemoryMB=7717
2023-04-18 12:55:44,211 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-18 12:55:44,212 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-18 12:55:44,406 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv101241596:84 , srv1019132701:84 , srv1033368439:84 , srv1033806304:84 , srv1067176924:84 , srv1073104064:84 , srv1086852629:84 , srv113036237:84 , srv1155718134:84 , srv1184341477:84 , srv1219989982:84 , srv1222828612:84 , srv1223704804:84 , srv1225851782:84 , srv1228653566:84 , srv1273691031:84 , srv1292963818:84 , srv1307885211:84 , srv133196864:84 , srv1338823943:84 , srv1352402846:84 , srv1354384389:84 , srv1374082991:84 , srv1382135708:84 , srv1386520133:84 , srv1387235061:84 , srv1397514252:84 , srv1404728050:84 , srv1420355908:84 , srv143222829:84 , srv1487965554:84 , srv1495184403:84 , srv1518367437:84 , srv1519250057:84 , srv1559714338:84 , srv1571076588:84 , srv1585620502:84 , srv1599969686:84 , srv1624957565:84 , srv1630133698:84 , srv1650026649:84 , srv1674080377:84 , srv1677960260:84 , srv1680218323:84 , srv1684013616:84 , srv1703071528:84 , srv1737609432:84 , srv1753696606:84 , srv1782180149:84 , srv1843460480:84 , srv186217414:84 , srv1884004171:84 , srv189832228:84 , srv1937385521:84 , srv1958956468:84 , srv1977549024:84 , srv1980510887:84 , srv1985607229:84 , srv2012271223:84 , srv2054810787:84 , srv2054848906:84 , srv2075176441:84 , srv208270427:84 , srv2087579612:84 , srv2094557345:84 , srv21477845:84 , srv273677004:84 , srv295017709:84 , srv300195378:84 , srv313988305:84 , srv3418747:84 , srv363022212:84 , srv377991526:84 , srv443697804:84 , srv455012141:84 , srv473255543:84 , srv503492308:84 , srv556795458:84 , srv572248745:84 , srv579849146:84 , srv604901465:84 , srv615545170:84 , srv62966372:84 , srv645350237:84 , srv649857010:84 , srv659033242:84 , srv677895085:84 , srv722547075:84 , srv760465917:84 , srv821271845:84 , srv826428247:84 , srv873043050:84 , srv87583748:84 , srv934956060:84 , srv944821146:84 , srv953482728:84 , srv966400395:84 , srv968354384:84 , srv981037547:84 , srv1030371365:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-04-18 12:55:44,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-18 12:55:44,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1518367437=33, srv760465917=89, srv2087579612=64, srv313988305=70, srv1222828612=12, srv1977549024=56, srv300195378=69, srv1677960260=43, srv473255543=76, srv968354384=98, srv443697804=74, srv873043050=92, srv1571076588=36, srv1958956468=55, srv953482728=96, srv604901465=81, srv1386520133=25, srv1684013616=45, srv1067176924=5, srv821271845=90, srv113036237=8, srv363022212=72, srv1404728050=28, srv1495184403=32, srv1703071528=46, srv1374082991=23, srv1273691031=16, srv503492308=77, srv455012141=75, srv981037547=99, srv62966372=83, srv1033806304=4, srv1884004171=52, srv1387235061=26, srv1737609432=47, srv3418747=71, srv1086852629=7, srv645350237=84, srv934956060=94, srv826428247=91, srv722547075=88, srv1225851782=14, srv1019132701=1, srv1223704804=13, srv1753696606=48, srv556795458=78, srv133196864=19, srv1487965554=31, srv1155718134=9, srv87583748=93, srv1650026649=41, srv649857010=85, srv944821146=95, srv186217414=51, srv1585620502=37, srv1292963818=17, srv1519250057=34, srv1219989982=11, srv2094557345=65, srv1599969686=38, srv2012271223=59, srv1420355908=29, srv1980510887=57, srv1624957565=39, srv1228653566=15, srv1937385521=54, srv273677004=67, srv1630133698=40, srv1674080377=42, srv579849146=80, srv1985607229=58, srv1397514252=27, srv1030371365=2, srv677895085=87, srv1382135708=24, srv189832228=53, srv615545170=82, srv208270427=63, srv1843460480=50, srv1073104064=6, srv1352402846=21, srv2054810787=60, srv1559714338=35, srv2054848906=61, srv1033368439=3, srv1307885211=18, srv1782180149=49, srv2075176441=62, srv1680218323=44, srv1338823943=20, srv572248745=79, srv377991526=73, srv1354384389=22, srv101241596=0, srv143222829=30, srv659033242=86, srv1184341477=10, srv21477845=66, srv295017709=68, srv966400395=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-18 12:55:44,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-18 12:55:44,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-18 12:55:44,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-18 12:55:44,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-18 12:55:44,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-18 12:55:44,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-18 12:55:44,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-18 12:55:44,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-18 12:55:44,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-18 12:55:44,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-18 12:55:44,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-18 12:55:44,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-18 12:55:44,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-18 12:55:44,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-18 12:55:44,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-18 12:55:44,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-18 12:55:44,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-18 12:55:44,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-18 12:55:44,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-18 12:55:44,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-18 12:55:44,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-18 12:55:44,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-18 12:55:44,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-18 12:55:44,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-18 12:55:44,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-18 12:55:44,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-18 12:55:44,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-18 12:55:44,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-18 12:55:44,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-18 12:55:44,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-18 12:55:44,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-18 12:55:44,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-18 12:55:44,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-18 12:55:44,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-18 12:55:44,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-18 12:55:44,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-18 12:55:44,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-18 12:55:44,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-18 12:55:44,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-18 12:55:44,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-18 12:55:44,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-18 12:55:44,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-18 12:55:44,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-18 12:55:44,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-18 12:55:44,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-18 12:55:44,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-18 12:55:44,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-18 12:55:44,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-18 12:55:44,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-18 12:55:44,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-18 12:55:44,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-18 12:55:44,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-18 12:55:44,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-18 12:55:44,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-18 12:55:44,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-18 12:55:44,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-18 12:55:44,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-18 12:55:44,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-18 12:55:44,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-18 12:55:44,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-18 12:55:44,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-18 12:55:44,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-18 12:55:44,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-18 12:55:44,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-18 12:55:44,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-18 12:55:44,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-18 12:55:44,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-18 12:55:44,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-18 12:55:44,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-18 12:55:44,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-18 12:55:44,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-18 12:55:44,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-18 12:55:44,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-18 12:55:44,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-18 12:55:44,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-18 12:55:44,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-18 12:55:44,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-18 12:55:44,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-18 12:55:44,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-18 12:55:44,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-18 12:55:44,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-18 12:55:44,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-18 12:55:44,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-18 12:55:44,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-18 12:55:44,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-18 12:55:44,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-18 12:55:44,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-18 12:55:44,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-18 12:55:44,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-18 12:55:44,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-18 12:55:44,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-18 12:55:44,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-18 12:55:44,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-18 12:55:44,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-18 12:55:44,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-18 12:55:44,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-18 12:55:44,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-18 12:55:44,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-18 12:55:44,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-18 12:55:44,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-18 12:55:44,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-18 12:55:44,908 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-18 12:55:44,909 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-18 12:55:44,909 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-18 12:55:45,940 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1296 ms to try 5000 different iterations.  Found a solution that moves 4677 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18023495216916294. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03893039311602312, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05765137688741345, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5196666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03893039311602312, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17583333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23833333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-18 12:55:45,985 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1487965554:82 , srv1518367437:82 , srv1843460480:82 , srv1958956468:82 , srv2054848906:82 , srv503492308:82 , srv649857010:82 , srv1228653566:83 , srv1292963818:83 , srv1674080377:83 , srv604901465:83 , srv760465917:83 , srv1677960260:84 , srv2012271223:84 , srv2075176441:84 , srv21477845:84 , srv295017709:84 , srv722547075:84 , srv968354384:84 , srv1033368439:85 , srv1067176924:85 , srv1086852629:85 , srv1273691031:85 , srv133196864:85 , srv1397514252:85 , srv1684013616:85 , srv1980510887:85 , srv208270427:85 , srv363022212:85 , srv377991526:85 , srv572248745:85 , srv645350237:85 , srv934956060:85 , srv113036237:86 , srv1219989982:86 , srv1352402846:86 , srv1354384389:86 , srv1374082991:86 , srv143222829:86 , srv1703071528:86 , srv1753696606:86 , srv1884004171:86 , srv189832228:86 , srv1977549024:86 , srv2094557345:86 , srv443697804:86 , srv615545170:86 , srv659033242:86 , srv677895085:86 , srv821271845:86 , srv953482728:86 , srv101241596:87 , srv1225851782:87 , srv1338823943:87 , srv1387235061:87 , srv1420355908:87 , srv1585620502:87 , srv1630133698:87 , srv1680218323:87 , srv1737609432:87 , srv186217414:87 , srv1937385521:87 , srv1985607229:87 , srv455012141:87 , srv87583748:87 , srv1155718134:88 , srv1222828612:88 , srv1223704804:88 , srv1382135708:88 , srv1386520133:88 , srv1571076588:88 , srv1624957565:88 , srv2054810787:88 , srv273677004:88 , srv313988305:88 , srv473255543:88 , srv579849146:88 , srv62966372:88 , srv944821146:88 , srv966400395:88 , srv981037547:88 , srv1033806304:89 , srv1073104064:89 , srv1184341477:89 , srv1404728050:89 , srv1495184403:89 , srv1519250057:89 , srv300195378:89 , srv873043050:89 , srv1559714338:90 , srv1599969686:90 , srv1650026649:90 , srv2087579612:90 , srv556795458:90 , srv1019132701:91 , srv1782180149:91 , srv826428247:91 , srv3418747:93 , srv1307885211:95 , srv1030371365:436 }
2023-04-18 12:55:45,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-18 12:55:45,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1518367437=33, srv760465917=89, srv2087579612=64, srv313988305=70, srv1222828612=12, srv1977549024=56, srv300195378=69, srv1677960260=43, srv473255543=76, srv968354384=98, srv443697804=74, srv873043050=92, srv1571076588=36, srv1958956468=55, srv953482728=96, srv604901465=81, srv1386520133=25, srv1684013616=45, srv1067176924=5, srv821271845=90, srv113036237=8, srv363022212=72, srv1404728050=28, srv1495184403=32, srv1703071528=46, srv1374082991=23, srv1273691031=16, srv503492308=77, srv455012141=75, srv981037547=99, srv62966372=83, srv1033806304=4, srv1884004171=52, srv1387235061=26, srv1737609432=47, srv3418747=71, srv1086852629=7, srv645350237=84, srv934956060=94, srv826428247=91, srv722547075=88, srv1225851782=14, srv1019132701=1, srv1223704804=13, srv1753696606=48, srv556795458=78, srv133196864=19, srv1487965554=31, srv1155718134=9, srv87583748=93, srv1650026649=41, srv649857010=85, srv944821146=95, srv186217414=51, srv1585620502=37, srv1292963818=17, srv1519250057=34, srv1219989982=11, srv2094557345=65, srv1599969686=38, srv2012271223=59, srv1420355908=29, srv1980510887=57, srv1624957565=39, srv1228653566=15, srv1937385521=54, srv273677004=67, srv1630133698=40, srv1674080377=42, srv579849146=80, srv1985607229=58, srv1397514252=27, srv1030371365=2, srv677895085=87, srv1382135708=24, srv189832228=53, srv615545170=82, srv208270427=63, srv1843460480=50, srv1073104064=6, srv1352402846=21, srv2054810787=60, srv1559714338=35, srv2054848906=61, srv1033368439=3, srv1307885211=18, srv1782180149=49, srv2075176441=62, srv1680218323=44, srv1338823943=20, srv572248745=79, srv377991526=73, srv1354384389=22, srv101241596=0, srv143222829=30, srv659033242=86, srv1184341477=10, srv21477845=66, srv295017709=68, srv966400395=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-18 12:55:46,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-18 12:55:46,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-18 12:55:46,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-18 12:55:46,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-18 12:55:46,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-18 12:55:46,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-18 12:55:46,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-18 12:55:46,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-18 12:55:46,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-18 12:55:46,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-18 12:55:46,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-18 12:55:46,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-18 12:55:46,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-18 12:55:46,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-18 12:55:46,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-18 12:55:46,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-18 12:55:46,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-18 12:55:46,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-18 12:55:46,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-18 12:55:46,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-18 12:55:46,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-18 12:55:46,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-18 12:55:46,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-18 12:55:46,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-18 12:55:46,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-18 12:55:46,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-18 12:55:46,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-18 12:55:46,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-18 12:55:46,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-18 12:55:46,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-18 12:55:46,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-18 12:55:46,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-18 12:55:46,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-18 12:55:46,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-18 12:55:46,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-18 12:55:46,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-18 12:55:46,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-18 12:55:46,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-18 12:55:46,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-18 12:55:46,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-18 12:55:46,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-18 12:55:46,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-18 12:55:46,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-18 12:55:46,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-18 12:55:46,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-18 12:55:46,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-18 12:55:46,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-18 12:55:46,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-18 12:55:46,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-18 12:55:46,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-18 12:55:46,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-18 12:55:46,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-18 12:55:46,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-18 12:55:46,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-18 12:55:46,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-18 12:55:46,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-18 12:55:46,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-18 12:55:46,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-18 12:55:46,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-18 12:55:46,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-18 12:55:46,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-18 12:55:46,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-18 12:55:46,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-18 12:55:46,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-18 12:55:46,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-18 12:55:46,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-18 12:55:46,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-18 12:55:46,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-18 12:55:46,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-18 12:55:46,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-18 12:55:46,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-18 12:55:46,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-18 12:55:46,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-18 12:55:46,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-18 12:55:46,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-18 12:55:46,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-18 12:55:46,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-18 12:55:46,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-18 12:55:46,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-18 12:55:46,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-18 12:55:46,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-18 12:55:46,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-18 12:55:46,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-18 12:55:46,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-18 12:55:46,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-18 12:55:46,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-18 12:55:46,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-18 12:55:46,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-18 12:55:46,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-18 12:55:46,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-18 12:55:46,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-18 12:55:46,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-18 12:55:46,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-18 12:55:46,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-18 12:55:46,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-18 12:55:46,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-18 12:55:46,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-18 12:55:46,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-18 12:55:46,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-18 12:55:46,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-18 12:55:46,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-18 12:55:46,213 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03893039311602312, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05765137688741345, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03893039311602312, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17583333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23833333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-18 12:55:46,213 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-18 12:55:46,213 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18020220311520338, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03893039311602312, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05765137688741345, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03893039311602312, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17583333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23833333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-18 12:55:46,637 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 570 ms to try 5000 different iterations.  Found a solution that moves 2671 regions; Going from a computed imbalance of 0.18020220311520338 to a new imbalance of 0.014483068762592016. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01991752542185); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.033671683461264604, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29677777777777775, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.01991752542185); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044583333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-18 12:55:46,656 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1307885211:78 , srv826428247:79 , srv3418747:80 , srv1782180149:81 , srv2087579612:81 , srv1019132701:82 , srv1386520133:82 , srv1404728050:82 , srv1624957565:82 , srv2054810787:82 , srv1223704804:83 , srv1382135708:83 , srv1495184403:83 , srv1559714338:83 , srv1571076588:83 , srv1599969686:83 , srv1650026649:83 , srv313988305:83 , srv579849146:83 , srv981037547:83 , srv1033806304:84 , srv1073104064:84 , srv113036237:84 , srv1184341477:84 , srv1219989982:84 , srv143222829:84 , srv1937385521:84 , srv300195378:84 , srv473255543:84 , srv556795458:84 , srv873043050:84 , srv944821146:84 , srv1222828612:85 , srv1338823943:85 , srv1519250057:85 , srv1680218323:85 , srv1737609432:85 , srv1884004171:85 , srv1977549024:85 , srv443697804:85 , srv572248745:85 , srv615545170:85 , srv62966372:85 , srv677895085:85 , srv821271845:85 , srv87583748:85 , srv953482728:85 , srv1155718134:86 , srv1352402846:86 , srv1374082991:86 , srv1420355908:86 , srv1585620502:86 , srv1630133698:86 , srv1753696606:86 , srv186217414:86 , srv189832228:86 , srv2094557345:86 , srv273677004:86 , srv377991526:86 , srv659033242:86 , srv966400395:86 , srv101241596:87 , srv1086852629:87 , srv1225851782:87 , srv1273691031:87 , srv1354384389:87 , srv1387235061:87 , srv1674080377:87 , srv1684013616:87 , srv1703071528:87 , srv1985607229:87 , srv2075176441:87 , srv208270427:87 , srv455012141:87 , srv934956060:87 , srv1033368439:88 , srv1067176924:88 , srv1292963818:88 , srv1397514252:88 , srv1518367437:88 , srv2012271223:88 , srv21477845:88 , srv363022212:88 , srv503492308:88 , srv722547075:88 , srv1677960260:89 , srv2054848906:89 , srv968354384:89 , srv1228653566:90 , srv133196864:90 , srv1843460480:90 , srv1958956468:90 , srv1980510887:90 , srv295017709:90 , srv604901465:90 , srv645350237:90 , srv649857010:91 , srv760465917:91 , srv1487965554:92 , srv1030371365:515 }
2023-04-18 12:55:46,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-18 12:55:46,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1518367437=33, srv760465917=89, srv2087579612=64, srv313988305=70, srv1222828612=12, srv1977549024=56, srv300195378=69, srv1677960260=43, srv473255543=76, srv968354384=98, srv443697804=74, srv873043050=92, srv1571076588=36, srv1958956468=55, srv953482728=96, srv604901465=81, srv1386520133=25, srv1684013616=45, srv1067176924=5, srv821271845=90, srv113036237=8, srv363022212=72, srv1404728050=28, srv1495184403=32, srv1703071528=46, srv1374082991=23, srv1273691031=16, srv503492308=77, srv455012141=75, srv981037547=99, srv62966372=83, srv1033806304=4, srv1884004171=52, srv1387235061=26, srv1737609432=47, srv3418747=71, srv1086852629=7, srv645350237=84, srv934956060=94, srv826428247=91, srv722547075=88, srv1225851782=14, srv1019132701=1, srv1223704804=13, srv1753696606=48, srv556795458=78, srv133196864=19, srv1487965554=31, srv1155718134=9, srv87583748=93, srv1650026649=41, srv649857010=85, srv944821146=95, srv186217414=51, srv1585620502=37, srv1292963818=17, srv1519250057=34, srv1219989982=11, srv2094557345=65, srv1599969686=38, srv2012271223=59, srv1420355908=29, srv1980510887=57, srv1624957565=39, srv1228653566=15, srv1937385521=54, srv273677004=67, srv1630133698=40, srv1674080377=42, srv579849146=80, srv1985607229=58, srv1397514252=27, srv1030371365=2, srv677895085=87, srv1382135708=24, srv189832228=53, srv615545170=82, srv208270427=63, srv1843460480=50, srv1073104064=6, srv1352402846=21, srv2054810787=60, srv1559714338=35, srv2054848906=61, srv1033368439=3, srv1307885211=18, srv1782180149=49, srv2075176441=62, srv1680218323=44, srv1338823943=20, srv572248745=79, srv377991526=73, srv1354384389=22, srv101241596=0, srv143222829=30, srv659033242=86, srv1184341477=10, srv21477845=66, srv295017709=68, srv966400395=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-18 12:55:46,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-18 12:55:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-18 12:55:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-18 12:55:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-18 12:55:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-18 12:55:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-18 12:55:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-18 12:55:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-18 12:55:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-18 12:55:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-18 12:55:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-18 12:55:46,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-18 12:55:46,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-18 12:55:46,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-18 12:55:46,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-18 12:55:46,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-18 12:55:46,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-18 12:55:46,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-18 12:55:46,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-18 12:55:46,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-18 12:55:46,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-18 12:55:46,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-18 12:55:46,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-18 12:55:46,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-18 12:55:46,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-18 12:55:46,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-18 12:55:46,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-18 12:55:46,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-18 12:55:46,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-18 12:55:46,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-18 12:55:46,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-18 12:55:46,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-18 12:55:46,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-18 12:55:46,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-18 12:55:46,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-18 12:55:46,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-18 12:55:46,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-18 12:55:46,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-18 12:55:46,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-18 12:55:46,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-18 12:55:46,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-18 12:55:46,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-18 12:55:46,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-18 12:55:46,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-18 12:55:46,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-18 12:55:46,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-18 12:55:46,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-18 12:55:46,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-18 12:55:46,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-18 12:55:46,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-18 12:55:46,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-18 12:55:46,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-18 12:55:46,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-18 12:55:46,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-18 12:55:46,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-18 12:55:46,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-18 12:55:46,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-18 12:55:46,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-18 12:55:46,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-18 12:55:46,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-18 12:55:46,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-18 12:55:46,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-18 12:55:46,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-18 12:55:46,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-18 12:55:46,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-18 12:55:46,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-18 12:55:46,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-18 12:55:46,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-18 12:55:46,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-18 12:55:46,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-18 12:55:46,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-18 12:55:46,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-18 12:55:46,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-18 12:55:46,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-18 12:55:46,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-18 12:55:46,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-18 12:55:46,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-18 12:55:46,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-18 12:55:46,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-18 12:55:46,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-18 12:55:46,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-18 12:55:46,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-18 12:55:46,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-18 12:55:46,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-18 12:55:46,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-18 12:55:46,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-18 12:55:46,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-18 12:55:46,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-18 12:55:46,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-18 12:55:46,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-18 12:55:46,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-18 12:55:46,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-18 12:55:46,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-18 12:55:46,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-18 12:55:46,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-18 12:55:46,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-18 12:55:46,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-18 12:55:46,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-18 12:55:46,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-18 12:55:46,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-18 12:55:46,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-18 12:55:46,857 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01991752542185); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.033671683461264604, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.01991752542185); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044583333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-18 12:55:46,858 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-18 12:55:46,858 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014464366020850303, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01991752542185); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.033671683461264604, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.01991752542185); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044583333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-18 12:55:47,255 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 533 ms to try 5000 different iterations.  Found a solution that moves 995 regions; Going from a computed imbalance of 0.014464366020850303 to a new imbalance of 7.442304231240783E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005088075744071); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00834174659477276); MoveCostFunction : (multiplier=7.0, imbalance=0.11055555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005088075744071); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-18 12:55:47,259 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv133196864:82 , srv645350237:82 , srv101241596:83 , srv1225851782:83 , srv1980510887:83 , srv556795458:83 , srv1019132701:84 , srv1033368439:84 , srv1033806304:84 , srv1067176924:84 , srv1073104064:84 , srv1155718134:84 , srv1184341477:84 , srv1307885211:84 , srv1387235061:84 , srv1487965554:84 , srv1519250057:84 , srv1585620502:84 , srv1599969686:84 , srv1677960260:84 , srv1703071528:84 , srv1985607229:84 , srv273677004:84 , srv295017709:84 , srv3418747:84 , srv363022212:84 , srv455012141:84 , srv760465917:84 , srv966400395:84 , srv1222828612:85 , srv1228653566:85 , srv1273691031:85 , srv1338823943:85 , srv1352402846:85 , srv1374082991:85 , srv1397514252:85 , srv1420355908:85 , srv1495184403:85 , srv1559714338:85 , srv1630133698:85 , srv1650026649:85 , srv1684013616:85 , srv1737609432:85 , srv1753696606:85 , srv186217414:85 , srv189832228:85 , srv2012271223:85 , srv208270427:85 , srv21477845:85 , srv604901465:85 , srv62966372:85 , srv649857010:85 , srv659033242:85 , srv873043050:85 , srv944821146:85 , srv968354384:85 , srv1086852629:86 , srv1292963818:86 , srv1354384389:86 , srv1382135708:86 , srv1571076588:86 , srv1680218323:86 , srv1782180149:86 , srv1843460480:86 , srv1884004171:86 , srv1937385521:86 , srv1958956468:86 , srv1977549024:86 , srv2054848906:86 , srv2094557345:86 , srv300195378:86 , srv473255543:86 , srv615545170:86 , srv722547075:86 , srv821271845:86 , srv87583748:86 , srv934956060:86 , srv981037547:86 , srv1219989982:87 , srv1223704804:87 , srv1386520133:87 , srv1404728050:87 , srv143222829:87 , srv1518367437:87 , srv1624957565:87 , srv2075176441:87 , srv2087579612:87 , srv313988305:87 , srv377991526:87 , srv443697804:87 , srv579849146:87 , srv677895085:87 , srv826428247:87 , srv113036237:88 , srv1674080377:88 , srv2054810787:88 , srv503492308:88 , srv572248745:88 , srv953482728:88 , srv1030371365:552 }
2023-04-18 12:55:47,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-18 12:55:47,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1518367437=33, srv760465917=89, srv2087579612=64, srv313988305=70, srv1222828612=12, srv1977549024=56, srv300195378=69, srv1677960260=43, srv473255543=76, srv968354384=98, srv443697804=74, srv873043050=92, srv1571076588=36, srv1958956468=55, srv953482728=96, srv604901465=81, srv1386520133=25, srv1684013616=45, srv1067176924=5, srv821271845=90, srv113036237=8, srv363022212=72, srv1404728050=28, srv1495184403=32, srv1703071528=46, srv1374082991=23, srv1273691031=16, srv503492308=77, srv455012141=75, srv981037547=99, srv62966372=83, srv1033806304=4, srv1884004171=52, srv1387235061=26, srv1737609432=47, srv3418747=71, srv1086852629=7, srv645350237=84, srv934956060=94, srv826428247=91, srv722547075=88, srv1225851782=14, srv1019132701=1, srv1223704804=13, srv1753696606=48, srv556795458=78, srv133196864=19, srv1487965554=31, srv1155718134=9, srv87583748=93, srv1650026649=41, srv649857010=85, srv944821146=95, srv186217414=51, srv1585620502=37, srv1292963818=17, srv1519250057=34, srv1219989982=11, srv2094557345=65, srv1599969686=38, srv2012271223=59, srv1420355908=29, srv1980510887=57, srv1624957565=39, srv1228653566=15, srv1937385521=54, srv273677004=67, srv1630133698=40, srv1674080377=42, srv579849146=80, srv1985607229=58, srv1397514252=27, srv1030371365=2, srv677895085=87, srv1382135708=24, srv189832228=53, srv615545170=82, srv208270427=63, srv1843460480=50, srv1073104064=6, srv1352402846=21, srv2054810787=60, srv1559714338=35, srv2054848906=61, srv1033368439=3, srv1307885211=18, srv1782180149=49, srv2075176441=62, srv1680218323=44, srv1338823943=20, srv572248745=79, srv377991526=73, srv1354384389=22, srv101241596=0, srv143222829=30, srv659033242=86, srv1184341477=10, srv21477845=66, srv295017709=68, srv966400395=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-18 12:55:47,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-18 12:55:47,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-18 12:55:47,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-18 12:55:47,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-18 12:55:47,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-18 12:55:47,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-18 12:55:47,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-18 12:55:47,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-18 12:55:47,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-18 12:55:47,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-18 12:55:47,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-18 12:55:47,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-18 12:55:47,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-18 12:55:47,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-18 12:55:47,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-18 12:55:47,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-18 12:55:47,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-18 12:55:47,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-18 12:55:47,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-18 12:55:47,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-18 12:55:47,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-18 12:55:47,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-18 12:55:47,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-18 12:55:47,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-18 12:55:47,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-18 12:55:47,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-18 12:55:47,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-18 12:55:47,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-18 12:55:47,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-18 12:55:47,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-18 12:55:47,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-18 12:55:47,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-18 12:55:47,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-18 12:55:47,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-18 12:55:47,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-18 12:55:47,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-18 12:55:47,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-18 12:55:47,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-18 12:55:47,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-18 12:55:47,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-18 12:55:47,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-18 12:55:47,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-18 12:55:47,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-18 12:55:47,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-18 12:55:47,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-18 12:55:47,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-18 12:55:47,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-18 12:55:47,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-18 12:55:47,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-18 12:55:47,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-18 12:55:47,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-18 12:55:47,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-18 12:55:47,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-18 12:55:47,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-18 12:55:47,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-18 12:55:47,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-18 12:55:47,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-18 12:55:47,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-18 12:55:47,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-18 12:55:47,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-18 12:55:47,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-18 12:55:47,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-18 12:55:47,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-18 12:55:47,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-18 12:55:47,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-18 12:55:47,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-18 12:55:47,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-18 12:55:47,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-18 12:55:47,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-18 12:55:47,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-18 12:55:47,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-18 12:55:47,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-18 12:55:47,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-18 12:55:47,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-18 12:55:47,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-18 12:55:47,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-18 12:55:47,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-18 12:55:47,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-18 12:55:47,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-18 12:55:47,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-18 12:55:47,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-18 12:55:47,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-18 12:55:47,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-18 12:55:47,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-18 12:55:47,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-18 12:55:47,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-18 12:55:47,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-18 12:55:47,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-18 12:55:47,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-18 12:55:47,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-18 12:55:47,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-18 12:55:47,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-18 12:55:47,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-18 12:55:47,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-18 12:55:47,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-18 12:55:47,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-18 12:55:47,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-18 12:55:47,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-18 12:55:47,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-18 12:55:47,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-18 12:55:47,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-18 12:55:47,466 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005088075744071); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00834174659477276); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005088075744071); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-18 12:55:47,466 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-18 12:55:47,466 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.372632842124325E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005088075744071); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00834174659477276); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005088075744071); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-18 12:55:47,788 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 460 ms to try 5000 different iterations.  Found a solution that moves 252 regions; Going from a computed imbalance of 7.372632842124325E-4 to a new imbalance of 3.6425963695509235E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.028, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-18 12:55:47,790 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1354384389:83 , srv1404728050:83 , srv1985607229:83 , srv300195378:83 , srv649857010:83 , srv953482728:83 , srv1086852629:84 , srv113036237:84 , srv1155718134:84 , srv1184341477:84 , srv1222828612:84 , srv1223704804:84 , srv1228653566:84 , srv1387235061:84 , srv1397514252:84 , srv1420355908:84 , srv1487965554:84 , srv1519250057:84 , srv1559714338:84 , srv1585620502:84 , srv1630133698:84 , srv1674080377:84 , srv1677960260:84 , srv1680218323:84 , srv1782180149:84 , srv1843460480:84 , srv186217414:84 , srv1958956468:84 , srv1980510887:84 , srv2054810787:84 , srv2075176441:84 , srv2087579612:84 , srv273677004:84 , srv295017709:84 , srv313988305:84 , srv377991526:84 , srv443697804:84 , srv455012141:84 , srv503492308:84 , srv572248745:84 , srv579849146:84 , srv604901465:84 , srv615545170:84 , srv62966372:84 , srv659033242:84 , srv677895085:84 , srv760465917:84 , srv873043050:84 , srv87583748:84 , srv934956060:84 , srv968354384:84 , srv1019132701:85 , srv1033368439:85 , srv1033806304:85 , srv1067176924:85 , srv1073104064:85 , srv1219989982:85 , srv1225851782:85 , srv1273691031:85 , srv1292963818:85 , srv1307885211:85 , srv133196864:85 , srv1338823943:85 , srv1352402846:85 , srv1374082991:85 , srv1382135708:85 , srv1386520133:85 , srv143222829:85 , srv1495184403:85 , srv1518367437:85 , srv1571076588:85 , srv1599969686:85 , srv1624957565:85 , srv1650026649:85 , srv1684013616:85 , srv1703071528:85 , srv1753696606:85 , srv1884004171:85 , srv189832228:85 , srv1937385521:85 , srv1977549024:85 , srv2012271223:85 , srv2054848906:85 , srv208270427:85 , srv2094557345:85 , srv21477845:85 , srv3418747:85 , srv363022212:85 , srv473255543:85 , srv556795458:85 , srv645350237:85 , srv722547075:85 , srv821271845:85 , srv826428247:85 , srv966400395:85 , srv981037547:85 , srv101241596:86 , srv1737609432:86 , srv944821146:86 , srv1030371365:639 }
2023-04-18 12:55:47,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-18 12:55:47,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1518367437=33, srv760465917=89, srv2087579612=64, srv313988305=70, srv1222828612=12, srv1977549024=56, srv300195378=69, srv1677960260=43, srv473255543=76, srv968354384=98, srv443697804=74, srv873043050=92, srv1571076588=36, srv1958956468=55, srv953482728=96, srv604901465=81, srv1386520133=25, srv1684013616=45, srv1067176924=5, srv821271845=90, srv113036237=8, srv363022212=72, srv1404728050=28, srv1495184403=32, srv1703071528=46, srv1374082991=23, srv1273691031=16, srv503492308=77, srv455012141=75, srv981037547=99, srv62966372=83, srv1033806304=4, srv1884004171=52, srv1387235061=26, srv1737609432=47, srv3418747=71, srv1086852629=7, srv645350237=84, srv934956060=94, srv826428247=91, srv722547075=88, srv1225851782=14, srv1019132701=1, srv1223704804=13, srv1753696606=48, srv556795458=78, srv133196864=19, srv1487965554=31, srv1155718134=9, srv87583748=93, srv1650026649=41, srv649857010=85, srv944821146=95, srv186217414=51, srv1585620502=37, srv1292963818=17, srv1519250057=34, srv1219989982=11, srv2094557345=65, srv1599969686=38, srv2012271223=59, srv1420355908=29, srv1980510887=57, srv1624957565=39, srv1228653566=15, srv1937385521=54, srv273677004=67, srv1630133698=40, srv1674080377=42, srv579849146=80, srv1985607229=58, srv1397514252=27, srv1030371365=2, srv677895085=87, srv1382135708=24, srv189832228=53, srv615545170=82, srv208270427=63, srv1843460480=50, srv1073104064=6, srv1352402846=21, srv2054810787=60, srv1559714338=35, srv2054848906=61, srv1033368439=3, srv1307885211=18, srv1782180149=49, srv2075176441=62, srv1680218323=44, srv1338823943=20, srv572248745=79, srv377991526=73, srv1354384389=22, srv101241596=0, srv143222829=30, srv659033242=86, srv1184341477=10, srv21477845=66, srv295017709=68, srv966400395=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-18 12:55:47,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-18 12:55:47,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-18 12:55:47,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-18 12:55:47,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-18 12:55:47,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-18 12:55:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-18 12:55:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-18 12:55:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-18 12:55:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-18 12:55:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-18 12:55:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-18 12:55:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-18 12:55:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-18 12:55:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-18 12:55:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-18 12:55:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-18 12:55:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-18 12:55:47,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-18 12:55:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-18 12:55:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-18 12:55:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-18 12:55:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-18 12:55:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-18 12:55:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-18 12:55:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-18 12:55:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-18 12:55:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-18 12:55:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-18 12:55:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-18 12:55:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-18 12:55:47,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-18 12:55:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-18 12:55:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-18 12:55:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-18 12:55:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-18 12:55:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-18 12:55:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-18 12:55:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-18 12:55:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-18 12:55:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-18 12:55:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-18 12:55:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-18 12:55:47,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-18 12:55:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-18 12:55:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-18 12:55:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-18 12:55:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-18 12:55:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-18 12:55:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-18 12:55:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-18 12:55:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-18 12:55:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-18 12:55:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-18 12:55:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-18 12:55:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-18 12:55:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-18 12:55:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-18 12:55:47,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-18 12:55:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-18 12:55:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-18 12:55:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-18 12:55:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-18 12:55:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-18 12:55:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-18 12:55:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-18 12:55:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-18 12:55:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-18 12:55:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-18 12:55:47,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-18 12:55:47,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-18 12:55:47,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-18 12:55:47,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-18 12:55:47,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-18 12:55:47,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-18 12:55:47,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-18 12:55:47,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-18 12:55:47,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-18 12:55:47,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-18 12:55:47,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-18 12:55:47,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-18 12:55:47,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-18 12:55:47,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-18 12:55:47,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-18 12:55:47,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-18 12:55:47,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-18 12:55:47,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-18 12:55:47,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-18 12:55:47,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-18 12:55:47,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-18 12:55:47,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-18 12:55:47,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-18 12:55:47,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-18 12:55:47,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-18 12:55:47,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-18 12:55:47,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-18 12:55:47,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-18 12:55:47,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-18 12:55:47,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-18 12:55:47,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-18 12:55:47,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-18 12:55:47,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-18 12:55:47,969 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-18 12:55:47,969 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-18 12:55:47,970 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.466142198120294E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-18 12:55:48,276 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 443 ms to try 5000 different iterations.  Found a solution that moves 34 regions; Going from a computed imbalance of 3.466142198120294E-5 to a new imbalance of 8.768720651464418E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.003777777777777778); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-18 12:55:48,277 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv101241596:83 , srv1650026649:83 , srv1737609432:83 , srv2094557345:83 , srv473255543:83 , srv722547075:83 , srv944821146:83 , srv966400395:83 , srv1019132701:84 , srv1033368439:84 , srv1033806304:84 , srv1067176924:84 , srv1073104064:84 , srv1086852629:84 , srv113036237:84 , srv1155718134:84 , srv1219989982:84 , srv1222828612:84 , srv1223704804:84 , srv1225851782:84 , srv1228653566:84 , srv1273691031:84 , srv1292963818:84 , srv1307885211:84 , srv133196864:84 , srv1338823943:84 , srv1352402846:84 , srv1354384389:84 , srv1374082991:84 , srv1382135708:84 , srv1386520133:84 , srv1387235061:84 , srv1397514252:84 , srv1420355908:84 , srv143222829:84 , srv1487965554:84 , srv1495184403:84 , srv1518367437:84 , srv1519250057:84 , srv1559714338:84 , srv1571076588:84 , srv1599969686:84 , srv1624957565:84 , srv1630133698:84 , srv1674080377:84 , srv1680218323:84 , srv1684013616:84 , srv1703071528:84 , srv1753696606:84 , srv1782180149:84 , srv1843460480:84 , srv186217414:84 , srv1884004171:84 , srv189832228:84 , srv1937385521:84 , srv1958956468:84 , srv1977549024:84 , srv1980510887:84 , srv2012271223:84 , srv2054810787:84 , srv2054848906:84 , srv2075176441:84 , srv208270427:84 , srv2087579612:84 , srv21477845:84 , srv273677004:84 , srv295017709:84 , srv300195378:84 , srv313988305:84 , srv3418747:84 , srv363022212:84 , srv377991526:84 , srv443697804:84 , srv455012141:84 , srv503492308:84 , srv556795458:84 , srv572248745:84 , srv579849146:84 , srv604901465:84 , srv62966372:84 , srv645350237:84 , srv677895085:84 , srv760465917:84 , srv821271845:84 , srv826428247:84 , srv873043050:84 , srv87583748:84 , srv934956060:84 , srv953482728:84 , srv968354384:84 , srv981037547:84 , srv1184341477:85 , srv1404728050:85 , srv1585620502:85 , srv1677960260:85 , srv1985607229:85 , srv615545170:85 , srv649857010:85 , srv659033242:85 , srv1030371365:684 }
2023-04-18 12:55:48,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-18 12:55:48,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1518367437=33, srv760465917=89, srv2087579612=64, srv313988305=70, srv1222828612=12, srv1977549024=56, srv300195378=69, srv1677960260=43, srv473255543=76, srv968354384=98, srv443697804=74, srv873043050=92, srv1571076588=36, srv1958956468=55, srv953482728=96, srv604901465=81, srv1386520133=25, srv1684013616=45, srv1067176924=5, srv821271845=90, srv113036237=8, srv363022212=72, srv1404728050=28, srv1495184403=32, srv1703071528=46, srv1374082991=23, srv1273691031=16, srv503492308=77, srv455012141=75, srv981037547=99, srv62966372=83, srv1033806304=4, srv1884004171=52, srv1387235061=26, srv1737609432=47, srv3418747=71, srv1086852629=7, srv645350237=84, srv934956060=94, srv826428247=91, srv722547075=88, srv1225851782=14, srv1019132701=1, srv1223704804=13, srv1753696606=48, srv556795458=78, srv133196864=19, srv1487965554=31, srv1155718134=9, srv87583748=93, srv1650026649=41, srv649857010=85, srv944821146=95, srv186217414=51, srv1585620502=37, srv1292963818=17, srv1519250057=34, srv1219989982=11, srv2094557345=65, srv1599969686=38, srv2012271223=59, srv1420355908=29, srv1980510887=57, srv1624957565=39, srv1228653566=15, srv1937385521=54, srv273677004=67, srv1630133698=40, srv1674080377=42, srv579849146=80, srv1985607229=58, srv1397514252=27, srv1030371365=2, srv677895085=87, srv1382135708=24, srv189832228=53, srv615545170=82, srv208270427=63, srv1843460480=50, srv1073104064=6, srv1352402846=21, srv2054810787=60, srv1559714338=35, srv2054848906=61, srv1033368439=3, srv1307885211=18, srv1782180149=49, srv2075176441=62, srv1680218323=44, srv1338823943=20, srv572248745=79, srv377991526=73, srv1354384389=22, srv101241596=0, srv143222829=30, srv659033242=86, srv1184341477=10, srv21477845=66, srv295017709=68, srv966400395=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-18 12:55:48,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-18 12:55:48,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-18 12:55:48,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-18 12:55:48,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-18 12:55:48,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-18 12:55:48,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-18 12:55:48,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-18 12:55:48,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-18 12:55:48,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-18 12:55:48,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-18 12:55:48,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-18 12:55:48,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-18 12:55:48,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-18 12:55:48,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-18 12:55:48,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-18 12:55:48,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-18 12:55:48,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-18 12:55:48,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-18 12:55:48,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-18 12:55:48,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-18 12:55:48,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-18 12:55:48,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-18 12:55:48,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-18 12:55:48,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-18 12:55:48,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-18 12:55:48,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-18 12:55:48,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-18 12:55:48,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-18 12:55:48,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-18 12:55:48,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-18 12:55:48,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-18 12:55:48,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-18 12:55:48,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-18 12:55:48,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-18 12:55:48,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-18 12:55:48,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-18 12:55:48,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-18 12:55:48,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-18 12:55:48,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-18 12:55:48,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-18 12:55:48,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-18 12:55:48,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-18 12:55:48,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-18 12:55:48,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-18 12:55:48,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-18 12:55:48,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-18 12:55:48,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-18 12:55:48,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-18 12:55:48,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-18 12:55:48,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-18 12:55:48,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-18 12:55:48,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-18 12:55:48,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-18 12:55:48,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-18 12:55:48,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-18 12:55:48,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-18 12:55:48,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-18 12:55:48,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-18 12:55:48,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-18 12:55:48,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-18 12:55:48,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-18 12:55:48,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-18 12:55:48,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-18 12:55:48,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-18 12:55:48,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-18 12:55:48,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-18 12:55:48,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-18 12:55:48,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-18 12:55:48,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-18 12:55:48,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-18 12:55:48,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-18 12:55:48,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-18 12:55:48,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-18 12:55:48,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-18 12:55:48,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-18 12:55:48,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-18 12:55:48,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-18 12:55:48,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-18 12:55:48,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-18 12:55:48,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-18 12:55:48,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-18 12:55:48,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-18 12:55:48,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-18 12:55:48,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-18 12:55:48,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-18 12:55:48,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-18 12:55:48,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-18 12:55:48,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-18 12:55:48,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-18 12:55:48,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-18 12:55:48,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-18 12:55:48,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-18 12:55:48,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-18 12:55:48,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-18 12:55:48,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-18 12:55:48,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-18 12:55:48,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-18 12:55:48,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-18 12:55:48,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-18 12:55:48,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-18 12:55:48,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-18 12:55:48,456 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=8.530647563026267E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-18 12:55:48,457 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv101241596:83 , srv1650026649:83 , srv1737609432:83 , srv2094557345:83 , srv473255543:83 , srv722547075:83 , srv944821146:83 , srv966400395:83 , srv1019132701:84 , srv1033368439:84 , srv1033806304:84 , srv1067176924:84 , srv1073104064:84 , srv1086852629:84 , srv113036237:84 , srv1155718134:84 , srv1219989982:84 , srv1222828612:84 , srv1223704804:84 , srv1225851782:84 , srv1228653566:84 , srv1273691031:84 , srv1292963818:84 , srv1307885211:84 , srv133196864:84 , srv1338823943:84 , srv1352402846:84 , srv1354384389:84 , srv1374082991:84 , srv1382135708:84 , srv1386520133:84 , srv1387235061:84 , srv1397514252:84 , srv1420355908:84 , srv143222829:84 , srv1487965554:84 , srv1495184403:84 , srv1518367437:84 , srv1519250057:84 , srv1559714338:84 , srv1571076588:84 , srv1599969686:84 , srv1624957565:84 , srv1630133698:84 , srv1674080377:84 , srv1680218323:84 , srv1684013616:84 , srv1703071528:84 , srv1753696606:84 , srv1782180149:84 , srv1843460480:84 , srv186217414:84 , srv1884004171:84 , srv189832228:84 , srv1937385521:84 , srv1958956468:84 , srv1977549024:84 , srv1980510887:84 , srv2012271223:84 , srv2054810787:84 , srv2054848906:84 , srv2075176441:84 , srv208270427:84 , srv2087579612:84 , srv21477845:84 , srv273677004:84 , srv295017709:84 , srv300195378:84 , srv313988305:84 , srv3418747:84 , srv363022212:84 , srv377991526:84 , srv443697804:84 , srv455012141:84 , srv503492308:84 , srv556795458:84 , srv572248745:84 , srv579849146:84 , srv604901465:84 , srv62966372:84 , srv645350237:84 , srv677895085:84 , srv760465917:84 , srv821271845:84 , srv826428247:84 , srv873043050:84 , srv87583748:84 , srv934956060:84 , srv953482728:84 , srv968354384:84 , srv981037547:84 , srv1184341477:85 , srv1404728050:85 , srv1585620502:85 , srv1677960260:85 , srv1985607229:85 , srv615545170:85 , srv649857010:85 , srv659033242:85 , srv1030371365:684 }
2023-04-18 12:55:48,490 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=426 (was 386) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=7197 (was 7717)
2023-04-18 12:55:48,499 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=426, ProcessCount=169, AvailableMemoryMB=7196
2023-04-18 12:55:48,501 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-18 12:55:48,502 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-04-18 12:55:48,513 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1045766790:84 , srv1196481130:84 , srv1241505232:84 , srv1277514093:84 , srv1298424906:84 , srv1492400407:84 , srv1544527582:84 , srv1572454418:84 , srv1581262289:84 , srv1669867095:84 , srv1738275144:84 , srv1796767114:84 , srv1963139699:84 , srv213485373:84 , srv231025369:84 , srv309161567:84 , srv402639060:84 , srv582480047:84 , srv653736040:84 , srv682906933:84 , srv764583229:84 , srv801346153:84 , srv838032980:84 , srv847948973:84 , srv866350835:84 , srv89872444:84 , srv913937636:84 , srv918482413:84 , srv987883978:84 , srv1058705669:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-04-18 12:55:48,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-18 12:55:48,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv213485373=14, srv1581262289=9, srv402639060=17, srv1277514093=4, srv1196481130=2, srv231025369=15, srv1492400407=6, srv1796767114=12, srv582480047=18, srv1669867095=10, srv1045766790=0, srv89872444=26, srv987883978=29, srv653736040=19, srv801346153=22, srv866350835=25, srv1298424906=5, srv309161567=16, srv1241505232=3, srv1963139699=13, srv918482413=28, srv1738275144=11, srv1572454418=8, srv1058705669=1, srv764583229=21, srv1544527582=7, srv838032980=23, srv682906933=20, srv847948973=24, srv913937636=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-18 12:55:48,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-18 12:55:48,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-18 12:55:48,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-18 12:55:48,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-18 12:55:48,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-18 12:55:48,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-18 12:55:48,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-18 12:55:48,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-18 12:55:48,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-18 12:55:48,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-18 12:55:48,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-18 12:55:48,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-18 12:55:48,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-18 12:55:48,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-18 12:55:48,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-18 12:55:48,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-18 12:55:48,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-18 12:55:48,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-18 12:55:48,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-18 12:55:48,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-18 12:55:48,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-18 12:55:48,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-18 12:55:48,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-18 12:55:48,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-18 12:55:48,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-18 12:55:48,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-18 12:55:48,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-18 12:55:48,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-18 12:55:48,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-18 12:55:48,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-18 12:55:48,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-18 12:55:48,554 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1243444417158235, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-18 12:55:48,555 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912584848539907, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1243444417158235, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-04-18 12:57:48,534 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6367619 different iterations.  Found a solution that moves 1866 regions; Going from a computed imbalance of 0.9912584848539907 to a new imbalance of 4.3553370884861656E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6911111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-18 12:57:48,540 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1045766790:90 , srv1058705669:90 , srv1196481130:90 , srv1241505232:90 , srv1277514093:90 , srv1298424906:90 , srv1492400407:90 , srv1544527582:90 , srv1572454418:90 , srv1581262289:90 , srv1669867095:90 , srv1738275144:90 , srv1796767114:90 , srv1963139699:90 , srv213485373:90 , srv231025369:90 , srv309161567:90 , srv402639060:90 , srv582480047:90 , srv653736040:90 , srv682906933:90 , srv764583229:90 , srv801346153:90 , srv838032980:90 , srv847948973:90 , srv866350835:90 , srv89872444:90 , srv913937636:90 , srv918482413:90 , srv987883978:90 }
2023-04-18 12:57:48,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-18 12:57:48,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv213485373=14, srv1581262289=9, srv402639060=17, srv1277514093=4, srv1196481130=2, srv231025369=15, srv1492400407=6, srv1796767114=12, srv582480047=18, srv1669867095=10, srv1045766790=0, srv89872444=26, srv987883978=29, srv653736040=19, srv801346153=22, srv866350835=25, srv1298424906=5, srv309161567=16, srv1241505232=3, srv1963139699=13, srv918482413=28, srv1738275144=11, srv1572454418=8, srv1058705669=1, srv764583229=21, srv1544527582=7, srv838032980=23, srv682906933=20, srv847948973=24, srv913937636=27} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-18 12:57:48,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-18 12:57:48,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-18 12:57:48,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-18 12:57:48,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-18 12:57:48,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-18 12:57:48,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-18 12:57:48,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-18 12:57:48,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-18 12:57:48,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-18 12:57:48,562 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-18 12:57:48,563 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1045766790:90 , srv1058705669:90 , srv1196481130:90 , srv1241505232:90 , srv1277514093:90 , srv1298424906:90 , srv1492400407:90 , srv1544527582:90 , srv1572454418:90 , srv1581262289:90 , srv1669867095:90 , srv1738275144:90 , srv1796767114:90 , srv1963139699:90 , srv213485373:90 , srv231025369:90 , srv309161567:90 , srv402639060:90 , srv582480047:90 , srv653736040:90 , srv682906933:90 , srv764583229:90 , srv801346153:90 , srv838032980:90 , srv847948973:90 , srv866350835:90 , srv89872444:90 , srv913937636:90 , srv918482413:90 , srv987883978:90 }
2023-04-18 12:57:48,578 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=246 (was 426), ProcessCount=169 (was 169), AvailableMemoryMB=7254 (was 7196) - AvailableMemoryMB LEAK? -