2024-03-07 10:53:49,680 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-03-07 10:53:49,929 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-03-07 10:53:50,022 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-03-07 10:53:50,083 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-07 10:53:50,089 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-07 10:53:50,102 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=362, ProcessCount=180, AvailableMemoryMB=7149
2024-03-07 10:53:50,104 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-07 10:53:50,104 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-07 10:53:50,221 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1007609841:84 , srv1022593018:84 , srv1035787790:84 , srv1069688197:84 , srv1070038904:84 , srv1080237637:84 , srv1115489141:84 , srv1132890378:84 , srv1160518701:84 , srv1184178808:84 , srv118613284:84 , srv1190107172:84 , srv1307861675:84 , srv1310104551:84 , srv1329360823:84 , srv1339498193:84 , srv1381740595:84 , srv1385351922:84 , srv1428197424:84 , srv1440643750:84 , srv1493579590:84 , srv1502933723:84 , srv1514462823:84 , srv1561793011:84 , srv1575187021:84 , srv159166280:84 , srv160158415:84 , srv1640969308:84 , srv1652345133:84 , srv1677236067:84 , srv1699444885:84 , srv172747859:84 , srv1730933020:84 , srv173534277:84 , srv1741437200:84 , srv1741892410:84 , srv176912145:84 , srv1799324283:84 , srv1799717010:84 , srv1817464860:84 , srv1825791773:84 , srv1903457306:84 , srv1909754344:84 , srv1932228283:84 , srv1969971098:84 , srv1991106332:84 , srv1991704661:84 , srv2075320889:84 , srv2098746084:84 , srv220209720:84 , srv231155890:84 , srv2324141:84 , srv257295369:84 , srv260612036:84 , srv277727494:84 , srv301466393:84 , srv303677705:84 , srv314940026:84 , srv330915297:84 , srv351737339:84 , srv354723352:84 , srv355186827:84 , srv367427114:84 , srv383297212:84 , srv386028384:84 , srv45445742:84 , srv476829767:84 , srv479410347:84 , srv482321505:84 , srv486940211:84 , srv564022528:84 , srv596049606:84 , srv635722391:84 , srv647375524:84 , srv655003079:84 , srv663781619:84 , srv668973620:84 , srv681852171:84 , srv708366612:84 , srv715683643:84 , srv716069043:84 , srv74647232:84 , srv748803494:84 , srv756030837:84 , srv760258847:84 , srv773713844:84 , srv79185730:84 , srv794142437:84 , srv845012583:84 , srv880858743:84 , srv888884439:84 , srv899634489:84 , srv90158790:84 , srv930885258:84 , srv974950669:84 , srv977131814:84 , srv981297459:84 , srv987406329:84 , srv996044287:84 , srv1998546990:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-03-07 10:53:50,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 10:53:50,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1817464860=39, srv1310104551=13, srv1677236067=29, srv383297212=64, srv708366612=79, srv79185730=87, srv1070038904=4, srv899634489=92, srv1160518701=8, srv1493579590=20, srv118613284=10, srv880858743=90, srv1069688197=3, srv1190107172=11, srv1640969308=27, srv2098746084=49, srv1730933020=32, srv1339498193=15, srv1132890378=7, srv159166280=25, srv1799717010=38, srv1799324283=37, srv930885258=94, srv748803494=83, srv1035787790=2, srv351737339=60, srv176912145=36, srv1575187021=24, srv668973620=77, srv301466393=56, srv635722391=73, srv173534277=33, srv596049606=72, srv715683643=80, srv1652345133=28, srv257295369=53, srv386028384=65, srv655003079=75, srv1385351922=17, srv277727494=55, srv1007609841=0, srv1502933723=21, srv2324141=52, srv773713844=86, srv794142437=88, srv74647232=82, srv845012583=89, srv303677705=57, srv330915297=59, srv231155890=51, srv160158415=26, srv1080237637=5, srv1307861675=12, srv1903457306=41, srv1514462823=22, srv974950669=95, srv564022528=71, srv1699444885=30, srv760258847=85, srv172747859=31, srv1022593018=1, srv1932228283=43, srv1741437200=34, srv90158790=93, srv1969971098=44, srv1115489141=6, srv1428197424=18, srv367427114=63, srv1561793011=23, srv486940211=70, srv716069043=81, srv977131814=96, srv1741892410=35, srv1909754344=42, srv2075320889=48, srv260612036=54, srv1440643750=19, srv45445742=66, srv479410347=68, srv996044287=99, srv1991106332=45, srv1184178808=9, srv987406329=98, srv476829767=67, srv888884439=91, srv663781619=76, srv355186827=62, srv756030837=84, srv647375524=74, srv482321505=69, srv1381740595=16, srv1998546990=47, srv354723352=61, srv1825791773=40, srv220209720=50, srv314940026=58, srv1329360823=14, srv1991704661=46, srv681852171=78, srv981297459=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 10:53:50,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 10:53:50,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 10:53:50,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 10:53:50,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 10:53:50,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 10:53:50,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 10:53:50,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 10:53:50,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 10:53:50,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 10:53:50,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 10:53:50,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 10:53:50,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 10:53:50,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 10:53:50,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 10:53:50,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 10:53:50,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 10:53:50,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 10:53:50,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 10:53:50,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 10:53:50,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 10:53:50,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 10:53:50,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 10:53:50,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 10:53:50,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 10:53:50,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 10:53:50,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 10:53:50,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 10:53:50,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 10:53:50,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 10:53:50,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 10:53:50,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-07 10:53:50,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-07 10:53:50,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-07 10:53:50,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-07 10:53:50,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-07 10:53:50,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-07 10:53:50,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-07 10:53:50,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-07 10:53:50,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-07 10:53:50,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-07 10:53:50,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-07 10:53:50,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-07 10:53:50,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-07 10:53:50,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-07 10:53:50,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-07 10:53:50,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-07 10:53:50,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-07 10:53:50,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-07 10:53:50,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-07 10:53:50,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-07 10:53:50,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-07 10:53:50,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-07 10:53:50,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-07 10:53:50,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-07 10:53:50,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-07 10:53:50,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-07 10:53:50,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-07 10:53:50,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-07 10:53:50,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-07 10:53:50,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-07 10:53:50,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-07 10:53:50,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-07 10:53:50,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-07 10:53:50,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-07 10:53:50,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-07 10:53:50,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-07 10:53:50,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-07 10:53:50,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-07 10:53:50,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-07 10:53:50,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-07 10:53:50,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-07 10:53:50,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-07 10:53:50,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-07 10:53:50,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-07 10:53:50,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-07 10:53:50,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-07 10:53:50,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-07 10:53:50,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-07 10:53:50,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-07 10:53:50,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-07 10:53:50,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-07 10:53:50,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-07 10:53:50,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-07 10:53:50,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-07 10:53:50,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-07 10:53:50,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-07 10:53:50,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-07 10:53:50,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-07 10:53:50,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-07 10:53:50,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-07 10:53:50,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-07 10:53:50,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-07 10:53:50,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-07 10:53:50,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-07 10:53:50,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-07 10:53:50,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-07 10:53:50,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-07 10:53:50,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-07 10:53:50,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-07 10:53:50,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-07 10:53:50,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-07 10:53:50,451 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 10:53:50,451 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-07 10:53:50,451 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-07 10:53:50,773 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 444 ms to try 5000 different iterations.  Found a solution that moves 4686 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1818967554845946. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04123907043436551, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05767667922652384, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5206666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04123907043436551, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17716666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24333333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 10:53:50,795 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1730933020:80 , srv756030837:80 , srv45445742:81 , srv708366612:81 , srv977131814:81 , srv1310104551:82 , srv173534277:82 , srv231155890:82 , srv260612036:82 , srv303677705:82 , srv383297212:82 , srv386028384:82 , srv899634489:82 , srv930885258:82 , srv1035787790:83 , srv1493579590:83 , srv2098746084:83 , srv479410347:83 , srv1070038904:84 , srv1160518701:84 , srv1184178808:84 , srv1575187021:84 , srv1825791773:84 , srv1903457306:84 , srv314940026:84 , srv635722391:84 , srv681852171:84 , srv74647232:84 , srv773713844:84 , srv79185730:84 , srv845012583:84 , srv1007609841:85 , srv1080237637:85 , srv1115489141:85 , srv1428197424:85 , srv1561793011:85 , srv1677236067:85 , srv1699444885:85 , srv1799717010:85 , srv2324141:85 , srv647375524:85 , srv1190107172:86 , srv1307861675:86 , srv1329360823:86 , srv1514462823:86 , srv160158415:86 , srv1652345133:86 , srv1741437200:86 , srv2075320889:86 , srv257295369:86 , srv355186827:86 , srv367427114:86 , srv482321505:86 , srv564022528:86 , srv668973620:86 , srv715683643:86 , srv716069043:86 , srv974950669:86 , srv1069688197:87 , srv118613284:87 , srv159166280:87 , srv1991106332:87 , srv301466393:87 , srv760258847:87 , srv90158790:87 , srv1132890378:88 , srv1339498193:88 , srv1799324283:88 , srv1991704661:88 , srv330915297:88 , srv351737339:88 , srv476829767:88 , srv596049606:88 , srv748803494:88 , srv880858743:88 , srv981297459:88 , srv987406329:88 , srv1381740595:89 , srv1741892410:89 , srv1909754344:89 , srv220209720:89 , srv354723352:89 , srv996044287:89 , srv176912145:90 , srv1817464860:90 , srv1969971098:90 , srv663781619:90 , srv794142437:90 , srv888884439:90 , srv277727494:91 , srv486940211:91 , srv1385351922:92 , srv1502933723:92 , srv1932228283:92 , srv1022593018:93 , srv1640969308:93 , srv1440643750:94 , srv172747859:95 , srv655003079:95 , srv1998546990:456 }
2024-03-07 10:53:50,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 10:53:50,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1817464860=39, srv1310104551=13, srv1677236067=29, srv383297212=64, srv708366612=79, srv79185730=87, srv1070038904=4, srv899634489=92, srv1160518701=8, srv1493579590=20, srv118613284=10, srv880858743=90, srv1069688197=3, srv1190107172=11, srv1640969308=27, srv2098746084=49, srv1730933020=32, srv1339498193=15, srv1132890378=7, srv159166280=25, srv1799717010=38, srv1799324283=37, srv930885258=94, srv748803494=83, srv1035787790=2, srv351737339=60, srv176912145=36, srv1575187021=24, srv668973620=77, srv301466393=56, srv635722391=73, srv173534277=33, srv596049606=72, srv715683643=80, srv1652345133=28, srv257295369=53, srv386028384=65, srv655003079=75, srv1385351922=17, srv277727494=55, srv1007609841=0, srv1502933723=21, srv2324141=52, srv773713844=86, srv794142437=88, srv74647232=82, srv845012583=89, srv303677705=57, srv330915297=59, srv231155890=51, srv160158415=26, srv1080237637=5, srv1307861675=12, srv1903457306=41, srv1514462823=22, srv974950669=95, srv564022528=71, srv1699444885=30, srv760258847=85, srv172747859=31, srv1022593018=1, srv1932228283=43, srv1741437200=34, srv90158790=93, srv1969971098=44, srv1115489141=6, srv1428197424=18, srv367427114=63, srv1561793011=23, srv486940211=70, srv716069043=81, srv977131814=96, srv1741892410=35, srv1909754344=42, srv2075320889=48, srv260612036=54, srv1440643750=19, srv45445742=66, srv479410347=68, srv996044287=99, srv1991106332=45, srv1184178808=9, srv987406329=98, srv476829767=67, srv888884439=91, srv663781619=76, srv355186827=62, srv756030837=84, srv647375524=74, srv482321505=69, srv1381740595=16, srv1998546990=47, srv354723352=61, srv1825791773=40, srv220209720=50, srv314940026=58, srv1329360823=14, srv1991704661=46, srv681852171=78, srv981297459=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 10:53:50,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 10:53:50,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 10:53:50,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 10:53:50,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 10:53:50,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 10:53:50,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 10:53:50,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 10:53:50,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 10:53:50,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 10:53:50,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 10:53:50,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 10:53:50,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 10:53:50,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 10:53:50,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 10:53:50,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 10:53:50,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 10:53:50,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 10:53:50,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 10:53:50,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 10:53:50,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 10:53:50,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 10:53:50,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 10:53:50,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 10:53:50,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 10:53:50,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 10:53:50,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 10:53:50,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 10:53:50,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 10:53:50,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 10:53:50,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 10:53:50,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-07 10:53:50,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-07 10:53:50,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-07 10:53:50,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-07 10:53:50,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-07 10:53:50,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-07 10:53:50,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-07 10:53:50,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-07 10:53:50,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-07 10:53:50,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-07 10:53:50,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-07 10:53:50,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-07 10:53:50,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-07 10:53:50,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-07 10:53:50,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-07 10:53:50,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-07 10:53:50,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-07 10:53:50,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-07 10:53:50,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-07 10:53:50,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-07 10:53:50,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-07 10:53:50,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-07 10:53:50,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-07 10:53:50,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-07 10:53:50,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-07 10:53:50,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-07 10:53:50,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-07 10:53:50,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-07 10:53:50,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-07 10:53:50,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-07 10:53:50,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-07 10:53:50,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-07 10:53:50,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-07 10:53:50,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-07 10:53:50,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-07 10:53:50,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-07 10:53:50,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-07 10:53:50,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-07 10:53:50,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-07 10:53:50,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-07 10:53:50,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-07 10:53:50,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-07 10:53:50,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-07 10:53:50,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-07 10:53:50,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-07 10:53:50,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-07 10:53:50,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-07 10:53:50,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-07 10:53:50,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-07 10:53:50,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-07 10:53:50,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-07 10:53:50,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-07 10:53:50,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-07 10:53:50,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-07 10:53:50,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-07 10:53:50,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-07 10:53:50,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-07 10:53:50,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-07 10:53:50,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-07 10:53:50,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-07 10:53:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-07 10:53:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-07 10:53:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-07 10:53:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-07 10:53:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-07 10:53:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-07 10:53:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-07 10:53:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-07 10:53:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-07 10:53:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-07 10:53:50,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-07 10:53:50,958 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04123907043436551, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05767667922652384, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04123907043436551, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17716666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24333333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 10:53:50,958 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-07 10:53:50,958 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1818639434112881, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04123907043436551, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05767667922652384, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04123907043436551, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17716666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24333333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-07 10:53:51,135 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 283 ms to try 5000 different iterations.  Found a solution that moves 2665 regions; Going from a computed imbalance of 0.1818639434112881 to a new imbalance of 0.015348962662491342. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02296133370998654); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0361192686978604, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2961111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02296133370998654); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043916666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 10:53:51,148 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv172747859:77 , srv1440643750:78 , srv1502933723:79 , srv1932228283:79 , srv1969971098:79 , srv1022593018:80 , srv1132890378:80 , srv486940211:80 , srv1381740595:81 , srv1640969308:81 , srv354723352:81 , srv655003079:81 , srv1385351922:82 , srv663781619:82 , srv1741892410:83 , srv1909754344:83 , srv1991704661:83 , srv367427114:83 , srv794142437:83 , srv987406329:83 , srv996044287:83 , srv1069688197:84 , srv160158415:84 , srv1652345133:84 , srv277727494:84 , srv330915297:84 , srv355186827:84 , srv476829767:84 , srv748803494:84 , srv760258847:84 , srv888884439:84 , srv1184178808:85 , srv118613284:85 , srv1190107172:85 , srv1514462823:85 , srv159166280:85 , srv1699444885:85 , srv1799324283:85 , srv1991106332:85 , srv220209720:85 , srv257295369:85 , srv351737339:85 , srv482321505:85 , srv596049606:85 , srv880858743:85 , srv1080237637:86 , srv1115489141:86 , srv1561793011:86 , srv1677236067:86 , srv1741437200:86 , srv176912145:86 , srv1799717010:86 , srv1817464860:86 , srv301466393:86 , srv314940026:86 , srv564022528:86 , srv715683643:86 , srv716069043:86 , srv773713844:86 , srv90158790:86 , srv974950669:86 , srv981297459:86 , srv1007609841:87 , srv1160518701:87 , srv1307861675:87 , srv1825791773:87 , srv1903457306:87 , srv2075320889:87 , srv647375524:87 , srv668973620:87 , srv681852171:87 , srv79185730:87 , srv1329360823:88 , srv1339498193:88 , srv1428197424:88 , srv1493579590:88 , srv1575187021:88 , srv2098746084:88 , srv2324141:88 , srv303677705:88 , srv635722391:88 , srv845012583:88 , srv1035787790:89 , srv1070038904:89 , srv1310104551:89 , srv173534277:89 , srv260612036:89 , srv383297212:89 , srv386028384:89 , srv479410347:89 , srv231155890:90 , srv74647232:90 , srv977131814:90 , srv708366612:91 , srv756030837:91 , srv899634489:91 , srv45445742:93 , srv930885258:93 , srv1730933020:94 , srv1998546990:522 }
2024-03-07 10:53:51,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 10:53:51,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1817464860=39, srv1310104551=13, srv1677236067=29, srv383297212=64, srv708366612=79, srv79185730=87, srv1070038904=4, srv899634489=92, srv1160518701=8, srv1493579590=20, srv118613284=10, srv880858743=90, srv1069688197=3, srv1190107172=11, srv1640969308=27, srv2098746084=49, srv1730933020=32, srv1339498193=15, srv1132890378=7, srv159166280=25, srv1799717010=38, srv1799324283=37, srv930885258=94, srv748803494=83, srv1035787790=2, srv351737339=60, srv176912145=36, srv1575187021=24, srv668973620=77, srv301466393=56, srv635722391=73, srv173534277=33, srv596049606=72, srv715683643=80, srv1652345133=28, srv257295369=53, srv386028384=65, srv655003079=75, srv1385351922=17, srv277727494=55, srv1007609841=0, srv1502933723=21, srv2324141=52, srv773713844=86, srv794142437=88, srv74647232=82, srv845012583=89, srv303677705=57, srv330915297=59, srv231155890=51, srv160158415=26, srv1080237637=5, srv1307861675=12, srv1903457306=41, srv1514462823=22, srv974950669=95, srv564022528=71, srv1699444885=30, srv760258847=85, srv172747859=31, srv1022593018=1, srv1932228283=43, srv1741437200=34, srv90158790=93, srv1969971098=44, srv1115489141=6, srv1428197424=18, srv367427114=63, srv1561793011=23, srv486940211=70, srv716069043=81, srv977131814=96, srv1741892410=35, srv1909754344=42, srv2075320889=48, srv260612036=54, srv1440643750=19, srv45445742=66, srv479410347=68, srv996044287=99, srv1991106332=45, srv1184178808=9, srv987406329=98, srv476829767=67, srv888884439=91, srv663781619=76, srv355186827=62, srv756030837=84, srv647375524=74, srv482321505=69, srv1381740595=16, srv1998546990=47, srv354723352=61, srv1825791773=40, srv220209720=50, srv314940026=58, srv1329360823=14, srv1991704661=46, srv681852171=78, srv981297459=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 10:53:51,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 10:53:51,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 10:53:51,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 10:53:51,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 10:53:51,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 10:53:51,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 10:53:51,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 10:53:51,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 10:53:51,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 10:53:51,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 10:53:51,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 10:53:51,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 10:53:51,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 10:53:51,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 10:53:51,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 10:53:51,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 10:53:51,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 10:53:51,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 10:53:51,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 10:53:51,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 10:53:51,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 10:53:51,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 10:53:51,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 10:53:51,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 10:53:51,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 10:53:51,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 10:53:51,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 10:53:51,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 10:53:51,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 10:53:51,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 10:53:51,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-07 10:53:51,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-07 10:53:51,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-07 10:53:51,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-07 10:53:51,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-07 10:53:51,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-07 10:53:51,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-07 10:53:51,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-07 10:53:51,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-07 10:53:51,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-07 10:53:51,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-07 10:53:51,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-07 10:53:51,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-07 10:53:51,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-07 10:53:51,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-07 10:53:51,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-07 10:53:51,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-07 10:53:51,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-07 10:53:51,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-07 10:53:51,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-07 10:53:51,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-07 10:53:51,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-07 10:53:51,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-07 10:53:51,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-07 10:53:51,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-07 10:53:51,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-07 10:53:51,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-07 10:53:51,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-07 10:53:51,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-07 10:53:51,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-07 10:53:51,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-07 10:53:51,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-07 10:53:51,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-07 10:53:51,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-07 10:53:51,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-07 10:53:51,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-07 10:53:51,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-07 10:53:51,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-07 10:53:51,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-07 10:53:51,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-07 10:53:51,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-07 10:53:51,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-07 10:53:51,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-07 10:53:51,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-07 10:53:51,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-07 10:53:51,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-07 10:53:51,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-07 10:53:51,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-07 10:53:51,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-07 10:53:51,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-07 10:53:51,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-07 10:53:51,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-07 10:53:51,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-07 10:53:51,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-07 10:53:51,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-07 10:53:51,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-07 10:53:51,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-07 10:53:51,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-07 10:53:51,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-07 10:53:51,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-07 10:53:51,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-07 10:53:51,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-07 10:53:51,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-07 10:53:51,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-07 10:53:51,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-07 10:53:51,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-07 10:53:51,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-07 10:53:51,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-07 10:53:51,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-07 10:53:51,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-07 10:53:51,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-07 10:53:51,265 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02296133370998654); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0361192686978604, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02296133370998654); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043916666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 10:53:51,265 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-07 10:53:51,265 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015330301933647586, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02296133370998654); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0361192686978604, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02296133370998654); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043916666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-07 10:53:51,414 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 217 ms to try 5000 different iterations.  Found a solution that moves 947 regions; Going from a computed imbalance of 0.015330301933647586 to a new imbalance of 7.654850057684076E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007446027270295803); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010615195681147192); MoveCostFunction : (multiplier=7.0, imbalance=0.10522222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007446027270295803); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 10:53:51,416 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1339498193:81 , srv655003079:81 , srv930885258:81 , srv176912145:82 , srv1817464860:82 , srv277727494:82 , srv888884439:82 , srv1329360823:83 , srv1385351922:83 , srv1640969308:83 , srv220209720:83 , srv1035787790:84 , srv1070038904:84 , srv1307861675:84 , srv1428197424:84 , srv1440643750:84 , srv1730933020:84 , srv1799324283:84 , srv1909754344:84 , srv2075320889:84 , srv45445742:84 , srv596049606:84 , srv74647232:84 , srv880858743:84 , srv899634489:84 , srv90158790:84 , srv981297459:84 , srv1007609841:85 , srv1022593018:85 , srv1069688197:85 , srv1190107172:85 , srv1575187021:85 , srv172747859:85 , srv1741437200:85 , srv1991106332:85 , srv1991704661:85 , srv2324141:85 , srv301466393:85 , srv330915297:85 , srv476829767:85 , srv635722391:85 , srv647375524:85 , srv663781619:85 , srv668973620:85 , srv708366612:85 , srv794142437:85 , srv996044287:85 , srv1115489141:86 , srv1160518701:86 , srv118613284:86 , srv1310104551:86 , srv1514462823:86 , srv1561793011:86 , srv159166280:86 , srv173534277:86 , srv1741892410:86 , srv1799717010:86 , srv1903457306:86 , srv1932228283:86 , srv2098746084:86 , srv231155890:86 , srv351737339:86 , srv383297212:86 , srv386028384:86 , srv482321505:86 , srv486940211:86 , srv564022528:86 , srv681852171:86 , srv715683643:86 , srv716069043:86 , srv748803494:86 , srv756030837:86 , srv760258847:86 , srv79185730:86 , srv845012583:86 , srv974950669:86 , srv977131814:86 , srv1080237637:87 , srv1381740595:87 , srv1493579590:87 , srv1502933723:87 , srv160158415:87 , srv1677236067:87 , srv1825791773:87 , srv260612036:87 , srv303677705:87 , srv314940026:87 , srv355186827:87 , srv479410347:87 , srv773713844:87 , srv987406329:87 , srv1652345133:88 , srv1699444885:88 , srv257295369:88 , srv354723352:88 , srv367427114:88 , srv1969971098:89 , srv1132890378:90 , srv1184178808:90 , srv1998546990:546 }
2024-03-07 10:53:51,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 10:53:51,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1817464860=39, srv1310104551=13, srv1677236067=29, srv383297212=64, srv708366612=79, srv79185730=87, srv1070038904=4, srv899634489=92, srv1160518701=8, srv1493579590=20, srv118613284=10, srv880858743=90, srv1069688197=3, srv1190107172=11, srv1640969308=27, srv2098746084=49, srv1730933020=32, srv1339498193=15, srv1132890378=7, srv159166280=25, srv1799717010=38, srv1799324283=37, srv930885258=94, srv748803494=83, srv1035787790=2, srv351737339=60, srv176912145=36, srv1575187021=24, srv668973620=77, srv301466393=56, srv635722391=73, srv173534277=33, srv596049606=72, srv715683643=80, srv1652345133=28, srv257295369=53, srv386028384=65, srv655003079=75, srv1385351922=17, srv277727494=55, srv1007609841=0, srv1502933723=21, srv2324141=52, srv773713844=86, srv794142437=88, srv74647232=82, srv845012583=89, srv303677705=57, srv330915297=59, srv231155890=51, srv160158415=26, srv1080237637=5, srv1307861675=12, srv1903457306=41, srv1514462823=22, srv974950669=95, srv564022528=71, srv1699444885=30, srv760258847=85, srv172747859=31, srv1022593018=1, srv1932228283=43, srv1741437200=34, srv90158790=93, srv1969971098=44, srv1115489141=6, srv1428197424=18, srv367427114=63, srv1561793011=23, srv486940211=70, srv716069043=81, srv977131814=96, srv1741892410=35, srv1909754344=42, srv2075320889=48, srv260612036=54, srv1440643750=19, srv45445742=66, srv479410347=68, srv996044287=99, srv1991106332=45, srv1184178808=9, srv987406329=98, srv476829767=67, srv888884439=91, srv663781619=76, srv355186827=62, srv756030837=84, srv647375524=74, srv482321505=69, srv1381740595=16, srv1998546990=47, srv354723352=61, srv1825791773=40, srv220209720=50, srv314940026=58, srv1329360823=14, srv1991704661=46, srv681852171=78, srv981297459=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 10:53:51,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 10:53:51,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 10:53:51,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 10:53:51,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 10:53:51,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 10:53:51,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 10:53:51,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 10:53:51,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 10:53:51,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 10:53:51,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 10:53:51,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 10:53:51,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 10:53:51,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 10:53:51,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 10:53:51,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 10:53:51,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 10:53:51,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 10:53:51,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 10:53:51,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 10:53:51,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 10:53:51,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 10:53:51,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 10:53:51,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 10:53:51,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 10:53:51,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 10:53:51,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 10:53:51,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 10:53:51,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 10:53:51,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 10:53:51,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 10:53:51,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-07 10:53:51,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-07 10:53:51,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-07 10:53:51,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-07 10:53:51,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-07 10:53:51,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-07 10:53:51,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-07 10:53:51,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-07 10:53:51,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-07 10:53:51,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-07 10:53:51,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-07 10:53:51,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-07 10:53:51,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-07 10:53:51,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-07 10:53:51,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-07 10:53:51,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-07 10:53:51,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-07 10:53:51,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-07 10:53:51,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-07 10:53:51,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-07 10:53:51,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-07 10:53:51,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-07 10:53:51,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-07 10:53:51,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-07 10:53:51,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-07 10:53:51,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-07 10:53:51,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-07 10:53:51,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-07 10:53:51,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-07 10:53:51,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-07 10:53:51,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-07 10:53:51,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-07 10:53:51,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-07 10:53:51,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-07 10:53:51,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-07 10:53:51,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-07 10:53:51,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-07 10:53:51,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-07 10:53:51,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-07 10:53:51,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-07 10:53:51,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-07 10:53:51,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-07 10:53:51,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-07 10:53:51,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-07 10:53:51,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-07 10:53:51,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-07 10:53:51,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-07 10:53:51,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-07 10:53:51,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-07 10:53:51,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-07 10:53:51,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-07 10:53:51,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-07 10:53:51,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-07 10:53:51,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-07 10:53:51,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-07 10:53:51,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-07 10:53:51,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-07 10:53:51,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-07 10:53:51,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-07 10:53:51,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-07 10:53:51,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-07 10:53:51,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-07 10:53:51,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-07 10:53:51,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-07 10:53:51,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-07 10:53:51,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-07 10:53:51,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-07 10:53:51,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-07 10:53:51,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-07 10:53:51,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-07 10:53:51,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-07 10:53:51,519 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007446027270295803); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010615195681147192); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007446027270295803); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 10:53:51,519 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-07 10:53:51,520 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.58853970040439E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007446027270295803); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010615195681147192); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007446027270295803); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-07 10:53:51,737 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 293 ms to try 5000 different iterations.  Found a solution that moves 319 regions; Going from a computed imbalance of 7.58853970040439E-4 to a new imbalance of 3.886029796417371E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002551379987942442); MoveCostFunction : (multiplier=7.0, imbalance=0.035444444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 10:53:51,738 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1132890378:83 , srv1184178808:83 , srv1817464860:83 , srv479410347:83 , srv748803494:83 , srv1022593018:84 , srv1080237637:84 , srv118613284:84 , srv1381740595:84 , srv1502933723:84 , srv1561793011:84 , srv159166280:84 , srv160158415:84 , srv1652345133:84 , srv1677236067:84 , srv1699444885:84 , srv172747859:84 , srv1741892410:84 , srv176912145:84 , srv1825791773:84 , srv1969971098:84 , srv231155890:84 , srv2324141:84 , srv257295369:84 , srv260612036:84 , srv301466393:84 , srv314940026:84 , srv351737339:84 , srv354723352:84 , srv355186827:84 , srv45445742:84 , srv564022528:84 , srv668973620:84 , srv74647232:84 , srv79185730:84 , srv794142437:84 , srv974950669:84 , srv981297459:84 , srv987406329:84 , srv1007609841:85 , srv1069688197:85 , srv1070038904:85 , srv1115489141:85 , srv1160518701:85 , srv1307861675:85 , srv1310104551:85 , srv1329360823:85 , srv1339498193:85 , srv1385351922:85 , srv1428197424:85 , srv1493579590:85 , srv1514462823:85 , srv1575187021:85 , srv1640969308:85 , srv1730933020:85 , srv173534277:85 , srv1741437200:85 , srv1799717010:85 , srv1932228283:85 , srv1991106332:85 , srv2098746084:85 , srv220209720:85 , srv277727494:85 , srv303677705:85 , srv330915297:85 , srv367427114:85 , srv383297212:85 , srv386028384:85 , srv476829767:85 , srv482321505:85 , srv596049606:85 , srv635722391:85 , srv647375524:85 , srv655003079:85 , srv663781619:85 , srv681852171:85 , srv708366612:85 , srv715683643:85 , srv716069043:85 , srv756030837:85 , srv760258847:85 , srv773713844:85 , srv845012583:85 , srv880858743:85 , srv888884439:85 , srv899634489:85 , srv90158790:85 , srv977131814:85 , srv996044287:85 , srv1035787790:86 , srv1190107172:86 , srv1440643750:86 , srv1799324283:86 , srv1903457306:86 , srv1991704661:86 , srv2075320889:86 , srv486940211:86 , srv930885258:86 , srv1909754344:87 , srv1998546990:618 }
2024-03-07 10:53:51,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 10:53:51,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1817464860=39, srv1310104551=13, srv1677236067=29, srv383297212=64, srv708366612=79, srv79185730=87, srv1070038904=4, srv899634489=92, srv1160518701=8, srv1493579590=20, srv118613284=10, srv880858743=90, srv1069688197=3, srv1190107172=11, srv1640969308=27, srv2098746084=49, srv1730933020=32, srv1339498193=15, srv1132890378=7, srv159166280=25, srv1799717010=38, srv1799324283=37, srv930885258=94, srv748803494=83, srv1035787790=2, srv351737339=60, srv176912145=36, srv1575187021=24, srv668973620=77, srv301466393=56, srv635722391=73, srv173534277=33, srv596049606=72, srv715683643=80, srv1652345133=28, srv257295369=53, srv386028384=65, srv655003079=75, srv1385351922=17, srv277727494=55, srv1007609841=0, srv1502933723=21, srv2324141=52, srv773713844=86, srv794142437=88, srv74647232=82, srv845012583=89, srv303677705=57, srv330915297=59, srv231155890=51, srv160158415=26, srv1080237637=5, srv1307861675=12, srv1903457306=41, srv1514462823=22, srv974950669=95, srv564022528=71, srv1699444885=30, srv760258847=85, srv172747859=31, srv1022593018=1, srv1932228283=43, srv1741437200=34, srv90158790=93, srv1969971098=44, srv1115489141=6, srv1428197424=18, srv367427114=63, srv1561793011=23, srv486940211=70, srv716069043=81, srv977131814=96, srv1741892410=35, srv1909754344=42, srv2075320889=48, srv260612036=54, srv1440643750=19, srv45445742=66, srv479410347=68, srv996044287=99, srv1991106332=45, srv1184178808=9, srv987406329=98, srv476829767=67, srv888884439=91, srv663781619=76, srv355186827=62, srv756030837=84, srv647375524=74, srv482321505=69, srv1381740595=16, srv1998546990=47, srv354723352=61, srv1825791773=40, srv220209720=50, srv314940026=58, srv1329360823=14, srv1991704661=46, srv681852171=78, srv981297459=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 10:53:51,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 10:53:51,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 10:53:51,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 10:53:51,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 10:53:51,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 10:53:51,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 10:53:51,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 10:53:51,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 10:53:51,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 10:53:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 10:53:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 10:53:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 10:53:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 10:53:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 10:53:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 10:53:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 10:53:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 10:53:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 10:53:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 10:53:51,752 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 10:53:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 10:53:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 10:53:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 10:53:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 10:53:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 10:53:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 10:53:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 10:53:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 10:53:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 10:53:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 10:53:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-07 10:53:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-07 10:53:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-07 10:53:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-07 10:53:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-07 10:53:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-07 10:53:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-07 10:53:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-07 10:53:51,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-07 10:53:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-07 10:53:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-07 10:53:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-07 10:53:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-07 10:53:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-07 10:53:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-07 10:53:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-07 10:53:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-07 10:53:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-07 10:53:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-07 10:53:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-07 10:53:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-07 10:53:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-07 10:53:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-07 10:53:51,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-07 10:53:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-07 10:53:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-07 10:53:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-07 10:53:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-07 10:53:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-07 10:53:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-07 10:53:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-07 10:53:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-07 10:53:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-07 10:53:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-07 10:53:51,755 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-07 10:53:51,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-07 10:53:51,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-07 10:53:51,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-07 10:53:51,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-07 10:53:51,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-07 10:53:51,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-07 10:53:51,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-07 10:53:51,756 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-07 10:53:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-07 10:53:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-07 10:53:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-07 10:53:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-07 10:53:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-07 10:53:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-07 10:53:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-07 10:53:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-07 10:53:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-07 10:53:51,757 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-07 10:53:51,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-07 10:53:51,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-07 10:53:51,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-07 10:53:51,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-07 10:53:51,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-07 10:53:51,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-07 10:53:51,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-07 10:53:51,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-07 10:53:51,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-07 10:53:51,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-07 10:53:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-07 10:53:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-07 10:53:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-07 10:53:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-07 10:53:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-07 10:53:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-07 10:53:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-07 10:53:51,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-07 10:53:51,836 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002551379987942442); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 10:53:51,837 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-07 10:53:51,837 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.6626612222651066E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002551379987942442); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-07 10:53:51,989 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 222 ms to try 5000 different iterations.  Found a solution that moves 38 regions; Going from a computed imbalance of 3.6626612222651066E-5 to a new imbalance of 9.314199796670486E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.004222222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 10:53:51,989 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1493579590:83 , srv1730933020:83 , srv1799324283:83 , srv1903457306:83 , srv1909754344:83 , srv2075320889:83 , srv257295369:83 , srv351737339:83 , srv486940211:83 , srv715683643:83 , srv716069043:83 , srv845012583:83 , srv1007609841:84 , srv1022593018:84 , srv1035787790:84 , srv1070038904:84 , srv1080237637:84 , srv1115489141:84 , srv1160518701:84 , srv1184178808:84 , srv118613284:84 , srv1190107172:84 , srv1307861675:84 , srv1310104551:84 , srv1329360823:84 , srv1339498193:84 , srv1385351922:84 , srv1428197424:84 , srv1440643750:84 , srv1502933723:84 , srv1514462823:84 , srv1575187021:84 , srv159166280:84 , srv1640969308:84 , srv1652345133:84 , srv1677236067:84 , srv1699444885:84 , srv173534277:84 , srv1741437200:84 , srv1741892410:84 , srv176912145:84 , srv1799717010:84 , srv1825791773:84 , srv1932228283:84 , srv1969971098:84 , srv1991106332:84 , srv1991704661:84 , srv2098746084:84 , srv220209720:84 , srv231155890:84 , srv2324141:84 , srv260612036:84 , srv277727494:84 , srv301466393:84 , srv303677705:84 , srv330915297:84 , srv354723352:84 , srv367427114:84 , srv383297212:84 , srv386028384:84 , srv45445742:84 , srv476829767:84 , srv479410347:84 , srv482321505:84 , srv564022528:84 , srv596049606:84 , srv635722391:84 , srv647375524:84 , srv655003079:84 , srv663781619:84 , srv668973620:84 , srv681852171:84 , srv708366612:84 , srv74647232:84 , srv756030837:84 , srv760258847:84 , srv773713844:84 , srv794142437:84 , srv880858743:84 , srv899634489:84 , srv90158790:84 , srv930885258:84 , srv974950669:84 , srv977131814:84 , srv981297459:84 , srv987406329:84 , srv996044287:84 , srv1069688197:85 , srv1132890378:85 , srv1381740595:85 , srv1561793011:85 , srv160158415:85 , srv172747859:85 , srv1817464860:85 , srv314940026:85 , srv355186827:85 , srv748803494:85 , srv79185730:85 , srv888884439:85 , srv1998546990:684 }
2024-03-07 10:53:51,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 10:53:51,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1817464860=39, srv1310104551=13, srv1677236067=29, srv383297212=64, srv708366612=79, srv79185730=87, srv1070038904=4, srv899634489=92, srv1160518701=8, srv1493579590=20, srv118613284=10, srv880858743=90, srv1069688197=3, srv1190107172=11, srv1640969308=27, srv2098746084=49, srv1730933020=32, srv1339498193=15, srv1132890378=7, srv159166280=25, srv1799717010=38, srv1799324283=37, srv930885258=94, srv748803494=83, srv1035787790=2, srv351737339=60, srv176912145=36, srv1575187021=24, srv668973620=77, srv301466393=56, srv635722391=73, srv173534277=33, srv596049606=72, srv715683643=80, srv1652345133=28, srv257295369=53, srv386028384=65, srv655003079=75, srv1385351922=17, srv277727494=55, srv1007609841=0, srv1502933723=21, srv2324141=52, srv773713844=86, srv794142437=88, srv74647232=82, srv845012583=89, srv303677705=57, srv330915297=59, srv231155890=51, srv160158415=26, srv1080237637=5, srv1307861675=12, srv1903457306=41, srv1514462823=22, srv974950669=95, srv564022528=71, srv1699444885=30, srv760258847=85, srv172747859=31, srv1022593018=1, srv1932228283=43, srv1741437200=34, srv90158790=93, srv1969971098=44, srv1115489141=6, srv1428197424=18, srv367427114=63, srv1561793011=23, srv486940211=70, srv716069043=81, srv977131814=96, srv1741892410=35, srv1909754344=42, srv2075320889=48, srv260612036=54, srv1440643750=19, srv45445742=66, srv479410347=68, srv996044287=99, srv1991106332=45, srv1184178808=9, srv987406329=98, srv476829767=67, srv888884439=91, srv663781619=76, srv355186827=62, srv756030837=84, srv647375524=74, srv482321505=69, srv1381740595=16, srv1998546990=47, srv354723352=61, srv1825791773=40, srv220209720=50, srv314940026=58, srv1329360823=14, srv1991704661=46, srv681852171=78, srv981297459=97} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 10:53:52,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 10:53:52,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 10:53:52,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 10:53:52,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 10:53:52,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 10:53:52,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 10:53:52,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 10:53:52,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 10:53:52,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 10:53:52,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 10:53:52,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 10:53:52,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 10:53:52,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 10:53:52,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 10:53:52,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 10:53:52,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 10:53:52,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 10:53:52,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 10:53:52,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 10:53:52,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 10:53:52,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 10:53:52,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 10:53:52,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 10:53:52,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 10:53:52,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 10:53:52,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 10:53:52,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 10:53:52,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 10:53:52,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 10:53:52,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 10:53:52,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-07 10:53:52,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-07 10:53:52,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-07 10:53:52,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-07 10:53:52,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-07 10:53:52,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-07 10:53:52,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-07 10:53:52,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-07 10:53:52,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-07 10:53:52,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-07 10:53:52,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-07 10:53:52,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-07 10:53:52,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-07 10:53:52,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-07 10:53:52,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-07 10:53:52,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-07 10:53:52,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-07 10:53:52,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-07 10:53:52,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-07 10:53:52,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-07 10:53:52,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-07 10:53:52,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-07 10:53:52,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-07 10:53:52,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-07 10:53:52,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-07 10:53:52,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-07 10:53:52,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-07 10:53:52,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-07 10:53:52,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-07 10:53:52,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-07 10:53:52,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-07 10:53:52,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-07 10:53:52,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-07 10:53:52,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-07 10:53:52,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-07 10:53:52,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-07 10:53:52,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-07 10:53:52,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-07 10:53:52,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-07 10:53:52,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-07 10:53:52,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-07 10:53:52,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-07 10:53:52,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-07 10:53:52,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-07 10:53:52,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-07 10:53:52,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-07 10:53:52,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-07 10:53:52,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-07 10:53:52,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-07 10:53:52,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-07 10:53:52,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-07 10:53:52,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-07 10:53:52,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-07 10:53:52,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-07 10:53:52,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-07 10:53:52,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-07 10:53:52,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-07 10:53:52,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-07 10:53:52,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-07 10:53:52,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-07 10:53:52,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-07 10:53:52,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-07 10:53:52,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-07 10:53:52,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-07 10:53:52,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-07 10:53:52,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-07 10:53:52,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-07 10:53:52,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-07 10:53:52,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-07 10:53:52,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-07 10:53:52,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-07 10:53:52,083 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=9.048118109592553E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 10:53:52,084 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1493579590:83 , srv1730933020:83 , srv1799324283:83 , srv1903457306:83 , srv1909754344:83 , srv2075320889:83 , srv257295369:83 , srv351737339:83 , srv486940211:83 , srv715683643:83 , srv716069043:83 , srv845012583:83 , srv1007609841:84 , srv1022593018:84 , srv1035787790:84 , srv1070038904:84 , srv1080237637:84 , srv1115489141:84 , srv1160518701:84 , srv1184178808:84 , srv118613284:84 , srv1190107172:84 , srv1307861675:84 , srv1310104551:84 , srv1329360823:84 , srv1339498193:84 , srv1385351922:84 , srv1428197424:84 , srv1440643750:84 , srv1502933723:84 , srv1514462823:84 , srv1575187021:84 , srv159166280:84 , srv1640969308:84 , srv1652345133:84 , srv1677236067:84 , srv1699444885:84 , srv173534277:84 , srv1741437200:84 , srv1741892410:84 , srv176912145:84 , srv1799717010:84 , srv1825791773:84 , srv1932228283:84 , srv1969971098:84 , srv1991106332:84 , srv1991704661:84 , srv2098746084:84 , srv220209720:84 , srv231155890:84 , srv2324141:84 , srv260612036:84 , srv277727494:84 , srv301466393:84 , srv303677705:84 , srv330915297:84 , srv354723352:84 , srv367427114:84 , srv383297212:84 , srv386028384:84 , srv45445742:84 , srv476829767:84 , srv479410347:84 , srv482321505:84 , srv564022528:84 , srv596049606:84 , srv635722391:84 , srv647375524:84 , srv655003079:84 , srv663781619:84 , srv668973620:84 , srv681852171:84 , srv708366612:84 , srv74647232:84 , srv756030837:84 , srv760258847:84 , srv773713844:84 , srv794142437:84 , srv880858743:84 , srv899634489:84 , srv90158790:84 , srv930885258:84 , srv974950669:84 , srv977131814:84 , srv981297459:84 , srv987406329:84 , srv996044287:84 , srv1069688197:85 , srv1132890378:85 , srv1381740595:85 , srv1561793011:85 , srv160158415:85 , srv172747859:85 , srv1817464860:85 , srv314940026:85 , srv355186827:85 , srv748803494:85 , srv79185730:85 , srv888884439:85 , srv1998546990:684 }
2024-03-07 10:53:52,106 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=341 (was 362), ProcessCount=178 (was 180), AvailableMemoryMB=6654 (was 7149)
2024-03-07 10:53:52,114 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=341, ProcessCount=178, AvailableMemoryMB=6654
2024-03-07 10:53:52,115 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-07 10:53:52,115 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-03-07 10:53:52,123 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1045384928:84 , srv1051666795:84 , srv1134665360:84 , srv1136785439:84 , srv1144710046:84 , srv1251858115:84 , srv1259745167:84 , srv1331821220:84 , srv1482506743:84 , srv1639044329:84 , srv1646369651:84 , srv1711213860:84 , srv1716781900:84 , srv1732362277:84 , srv1802165293:84 , srv1841257564:84 , srv2010316736:84 , srv2066928661:84 , srv2082948716:84 , srv245068779:84 , srv283931730:84 , srv307894221:84 , srv371629661:84 , srv620071732:84 , srv748782406:84 , srv778580170:84 , srv908185420:84 , srv923951647:84 , srv951502521:84 , srv1008455736:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-03-07 10:53:52,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 10:53:52,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv951502521=29, srv1259745167=7, srv1482506743=9, srv1639044329=10, srv283931730=21, srv748782406=25, srv1251858115=6, srv1144710046=5, srv1136785439=4, srv1716781900=13, srv371629661=23, srv1051666795=2, srv1045384928=1, srv2082948716=19, srv1732362277=14, srv2010316736=17, srv1134665360=3, srv1646369651=11, srv620071732=24, srv908185420=27, srv778580170=26, srv1802165293=15, srv1841257564=16, srv1331821220=8, srv1711213860=12, srv2066928661=18, srv307894221=22, srv245068779=20, srv923951647=28, srv1008455736=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 10:53:52,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 10:53:52,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 10:53:52,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 10:53:52,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 10:53:52,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 10:53:52,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 10:53:52,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 10:53:52,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 10:53:52,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 10:53:52,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 10:53:52,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 10:53:52,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 10:53:52,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 10:53:52,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 10:53:52,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 10:53:52,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 10:53:52,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 10:53:52,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 10:53:52,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 10:53:52,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 10:53:52,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 10:53:52,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 10:53:52,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 10:53:52,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 10:53:52,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 10:53:52,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 10:53:52,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 10:53:52,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 10:53:52,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 10:53:52,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 10:53:52,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-07 10:53:52,146 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1546920362053354, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 10:53:52,147 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912680472819203, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1546920362053354, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-03-07 10:55:52,137 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9616492 different iterations.  Found a solution that moves 1827 regions; Going from a computed imbalance of 0.9912680472819203 to a new imbalance of 4.264309142906872E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6766666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 10:55:52,142 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1008455736:90 , srv1045384928:90 , srv1051666795:90 , srv1134665360:90 , srv1136785439:90 , srv1144710046:90 , srv1251858115:90 , srv1259745167:90 , srv1331821220:90 , srv1482506743:90 , srv1639044329:90 , srv1646369651:90 , srv1711213860:90 , srv1716781900:90 , srv1732362277:90 , srv1802165293:90 , srv1841257564:90 , srv2010316736:90 , srv2066928661:90 , srv2082948716:90 , srv245068779:90 , srv283931730:90 , srv307894221:90 , srv371629661:90 , srv620071732:90 , srv748782406:90 , srv778580170:90 , srv908185420:90 , srv923951647:90 , srv951502521:90 }
2024-03-07 10:55:52,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 10:55:52,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv951502521=29, srv1259745167=7, srv1482506743=9, srv1639044329=10, srv283931730=21, srv748782406=25, srv1251858115=6, srv1144710046=5, srv1136785439=4, srv1716781900=13, srv371629661=23, srv1051666795=2, srv1045384928=1, srv2082948716=19, srv1732362277=14, srv2010316736=17, srv1134665360=3, srv1646369651=11, srv620071732=24, srv908185420=27, srv778580170=26, srv1802165293=15, srv1841257564=16, srv1331821220=8, srv1711213860=12, srv2066928661=18, srv307894221=22, srv245068779=20, srv923951647=28, srv1008455736=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 10:55:52,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 10:55:52,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 10:55:52,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 10:55:52,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 10:55:52,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 10:55:52,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 10:55:52,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 10:55:52,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 10:55:52,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 10:55:52,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 10:55:52,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 10:55:52,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 10:55:52,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 10:55:52,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 10:55:52,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 10:55:52,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 10:55:52,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 10:55:52,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 10:55:52,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 10:55:52,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 10:55:52,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 10:55:52,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 10:55:52,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 10:55:52,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 10:55:52,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 10:55:52,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 10:55:52,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 10:55:52,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 10:55:52,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 10:55:52,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 10:55:52,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-07 10:55:52,157 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 10:55:52,157 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1008455736:90 , srv1045384928:90 , srv1051666795:90 , srv1134665360:90 , srv1136785439:90 , srv1144710046:90 , srv1251858115:90 , srv1259745167:90 , srv1331821220:90 , srv1482506743:90 , srv1639044329:90 , srv1646369651:90 , srv1711213860:90 , srv1716781900:90 , srv1732362277:90 , srv1802165293:90 , srv1841257564:90 , srv2010316736:90 , srv2066928661:90 , srv2082948716:90 , srv245068779:90 , srv283931730:90 , srv307894221:90 , srv371629661:90 , srv620071732:90 , srv748782406:90 , srv778580170:90 , srv908185420:90 , srv923951647:90 , srv951502521:90 }
2024-03-07 10:55:52,172 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=173 (was 341), ProcessCount=177 (was 178), AvailableMemoryMB=6327 (was 6654)