2024-02-10 10:53:07,220 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-02-10 10:53:07,518 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-02-10 10:53:07,629 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-02-10 10:53:07,687 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-02-10 10:53:07,693 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-02-10 10:53:07,706 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=248, ProcessCount=167, AvailableMemoryMB=4714
2024-02-10 10:53:07,708 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-02-10 10:53:07,708 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-02-10 10:53:07,843 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv101335444:84 , srv1014695832:84 , srv1028201144:84 , srv1043701192:84 , srv1055000512:84 , srv1064588497:84 , srv1070843140:84 , srv1080454172:84 , srv1086704446:84 , srv1099379848:84 , srv1124788687:84 , srv1165663999:84 , srv1178809611:84 , srv1192211390:84 , srv1207435503:84 , srv1216505715:84 , srv1227514126:84 , srv1248340678:84 , srv1254857429:84 , srv1258672475:84 , srv1286505947:84 , srv1311907108:84 , srv1324055188:84 , srv1328979338:84 , srv1330246072:84 , srv1334464643:84 , srv1336406:84 , srv1366279572:84 , srv1401539833:84 , srv1416194945:84 , srv1441754313:84 , srv1481650629:84 , srv1529451536:84 , srv1544344181:84 , srv1559590001:84 , srv1588504420:84 , srv1588820842:84 , srv16045101:84 , srv1622176542:84 , srv1635892467:84 , srv1663635688:84 , srv1664820251:84 , srv1688997183:84 , srv1692054239:84 , srv1718243420:84 , srv172504275:84 , srv1745418923:84 , srv1769629450:84 , srv1816229645:84 , srv1847094714:84 , srv1866176027:84 , srv1868326145:84 , srv1880977814:84 , srv1905238773:84 , srv1906045865:84 , srv190895191:84 , srv1933124178:84 , srv2020068472:84 , srv2049287243:84 , srv205372954:84 , srv2059371526:84 , srv2059893869:84 , srv2068079076:84 , srv2119563540:84 , srv2121270026:84 , srv2144453205:84 , srv247280782:84 , srv251907002:84 , srv261590855:84 , srv281469621:84 , srv303305410:84 , srv306781200:84 , srv311244598:84 , srv316203919:84 , srv342301478:84 , srv394080521:84 , srv397774068:84 , srv400924074:84 , srv414568268:84 , srv436710960:84 , srv454119414:84 , srv461548863:84 , srv475479231:84 , srv479658305:84 , srv515198387:84 , srv548587880:84 , srv582344527:84 , srv589040864:84 , srv607878581:84 , srv644535686:84 , srv695020699:84 , srv72904101:84 , srv740684454:84 , srv750776119:84 , srv863800798:84 , srv865441434:84 , srv868227428:84 , srv892940418:84 , srv953891198:84 , srv1196961439:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-02-10 10:53:07,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-10 10:53:07,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2068079076=63, srv953891198=99, srv1880977814=53, srv1401539833=29, srv868227428=97, srv1718243420=45, srv1745418923=47, srv2049287243=59, srv281469621=70, srv205372954=60, srv740684454=93, srv306781200=72, srv316203919=74, srv1080454172=7, srv394080521=76, srv582344527=87, srv1816229645=49, srv589040864=88, srv454119414=81, srv461548863=82, srv1330246072=25, srv2121270026=65, srv1311907108=22, srv72904101=92, srv1664820251=42, srv1014695832=1, srv548587880=86, srv303305410=71, srv1366279572=28, srv1905238773=54, srv311244598=73, srv1481650629=32, srv1028201144=2, srv1043701192=3, srv644535686=90, srv607878581=89, srv1588504420=36, srv863800798=95, srv251907002=68, srv1099379848=9, srv247280782=67, srv397774068=77, srv2059371526=61, srv1635892467=40, srv1165663999=11, srv1416194945=30, srv1692054239=44, srv16045101=38, srv1588820842=37, srv1769629450=48, srv1336406=27, srv1334464643=26, srv1529451536=33, srv1086704446=8, srv1227514126=17, srv1324055188=23, srv1559590001=35, srv2144453205=66, srv750776119=94, srv892940418=98, srv1196961439=14, srv1070843140=6, srv1258672475=20, srv190895191=56, srv695020699=91, srv414568268=79, srv2020068472=58, srv1933124178=57, srv342301478=75, srv1124788687=10, srv2119563540=64, srv1622176542=39, srv172504275=46, srv865441434=96, srv1216505715=16, srv2059893869=62, srv1663635688=41, srv1248340678=18, srv101335444=0, srv479658305=84, srv1688997183=43, srv1847094714=50, srv1441754313=31, srv1286505947=21, srv1254857429=19, srv1868326145=52, srv1906045865=55, srv1207435503=15, srv1192211390=13, srv436710960=80, srv400924074=78, srv1544344181=34, srv1328979338=24, srv515198387=85, srv1055000512=4, srv1866176027=51, srv1064588497=5, srv1178809611=12, srv261590855=69, srv475479231=83} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-10 10:53:07,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-10 10:53:07,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-10 10:53:07,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-10 10:53:07,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-10 10:53:07,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-10 10:53:07,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-10 10:53:07,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-10 10:53:07,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-10 10:53:07,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-10 10:53:07,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-10 10:53:07,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-10 10:53:07,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-10 10:53:07,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-10 10:53:07,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-10 10:53:07,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-10 10:53:07,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-10 10:53:07,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-10 10:53:07,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-10 10:53:07,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-10 10:53:07,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-10 10:53:07,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-10 10:53:07,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-10 10:53:07,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-10 10:53:07,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-10 10:53:07,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-10 10:53:07,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-10 10:53:07,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-10 10:53:07,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-10 10:53:07,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-10 10:53:07,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-10 10:53:07,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-10 10:53:07,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-10 10:53:07,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-10 10:53:07,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-10 10:53:07,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-10 10:53:07,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-10 10:53:07,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-10 10:53:07,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-10 10:53:07,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-10 10:53:07,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-10 10:53:07,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-10 10:53:07,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-10 10:53:07,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-10 10:53:07,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-10 10:53:07,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-10 10:53:07,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-10 10:53:07,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-10 10:53:07,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-10 10:53:07,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-10 10:53:07,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-10 10:53:07,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-10 10:53:07,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-10 10:53:07,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-10 10:53:07,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-10 10:53:07,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-10 10:53:07,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-10 10:53:07,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-10 10:53:07,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-10 10:53:07,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-10 10:53:07,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-10 10:53:07,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-10 10:53:07,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-10 10:53:07,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-10 10:53:07,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-10 10:53:07,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-10 10:53:07,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-10 10:53:07,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-10 10:53:07,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-10 10:53:07,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-10 10:53:07,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-10 10:53:07,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-10 10:53:07,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-10 10:53:07,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-10 10:53:07,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-10 10:53:07,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-10 10:53:07,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-10 10:53:07,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-10 10:53:07,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-10 10:53:07,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-10 10:53:07,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-10 10:53:07,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-10 10:53:07,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-10 10:53:07,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-10 10:53:07,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-10 10:53:07,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-10 10:53:07,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-10 10:53:07,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-10 10:53:07,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-10 10:53:07,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-10 10:53:07,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-10 10:53:07,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-10 10:53:07,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-10 10:53:07,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-10 10:53:07,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-10 10:53:07,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-10 10:53:07,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-10 10:53:07,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-10 10:53:07,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-10 10:53:07,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-10 10:53:07,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-10 10:53:07,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-10 10:53:08,258 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-10 10:53:08,259 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-10 10:53:08,259 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-10 10:53:08,669 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 666 ms to try 5000 different iterations.  Found a solution that moves 4712 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1785577142693209. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04075145252507087, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05804329047377973, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5235555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04075145252507087, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1735, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24291666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-10 10:53:08,688 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1124788687:82 , srv1311907108:82 , srv1622176542:82 , srv1663635688:82 , srv414568268:82 , srv436710960:82 , srv589040864:82 , srv607878581:82 , srv1028201144:83 , srv1258672475:83 , srv1366279572:83 , srv1588504420:83 , srv1692054239:83 , srv1745418923:83 , srv2119563540:83 , srv695020699:83 , srv865441434:83 , srv953891198:83 , srv1043701192:84 , srv1165663999:84 , srv1248340678:84 , srv1718243420:84 , srv205372954:84 , srv394080521:84 , srv515198387:84 , srv1099379848:85 , srv1254857429:85 , srv1328979338:85 , srv1336406:85 , srv1529451536:85 , srv1635892467:85 , srv1688997183:85 , srv1866176027:85 , srv1880977814:85 , srv2020068472:85 , srv303305410:85 , srv397774068:85 , srv475479231:85 , srv644535686:85 , srv1080454172:86 , srv1086704446:86 , srv1192211390:86 , srv1207435503:86 , srv1227514126:86 , srv1286505947:86 , srv1330246072:86 , srv1441754313:86 , srv1769629450:86 , srv1868326145:86 , srv2059893869:86 , srv247280782:86 , srv261590855:86 , srv281469621:86 , srv311244598:86 , srv342301478:86 , srv101335444:87 , srv1324055188:87 , srv1416194945:87 , srv1559590001:87 , srv1847094714:87 , srv1906045865:87 , srv2049287243:87 , srv2059371526:87 , srv251907002:87 , srv454119414:87 , srv461548863:87 , srv479658305:87 , srv548587880:87 , srv582344527:87 , srv750776119:87 , srv863800798:87 , srv1178809611:88 , srv1216505715:88 , srv1544344181:88 , srv1588820842:88 , srv172504275:88 , srv1816229645:88 , srv72904101:88 , srv740684454:88 , srv1014695832:89 , srv1064588497:89 , srv1070843140:89 , srv1481650629:89 , srv190895191:89 , srv2144453205:89 , srv1334464643:90 , srv1401539833:90 , srv1905238773:90 , srv1933124178:90 , srv2068079076:90 , srv2121270026:90 , srv868227428:90 , srv1055000512:91 , srv16045101:91 , srv306781200:91 , srv1664820251:92 , srv316203919:92 , srv400924074:93 , srv892940418:98 , srv1196961439:452 }
2024-02-10 10:53:08,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-10 10:53:08,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2068079076=63, srv953891198=99, srv1880977814=53, srv1401539833=29, srv868227428=97, srv1718243420=45, srv1745418923=47, srv2049287243=59, srv281469621=70, srv205372954=60, srv740684454=93, srv306781200=72, srv316203919=74, srv1080454172=7, srv394080521=76, srv582344527=87, srv1816229645=49, srv589040864=88, srv454119414=81, srv461548863=82, srv1330246072=25, srv2121270026=65, srv1311907108=22, srv72904101=92, srv1664820251=42, srv1014695832=1, srv548587880=86, srv303305410=71, srv1366279572=28, srv1905238773=54, srv311244598=73, srv1481650629=32, srv1028201144=2, srv1043701192=3, srv644535686=90, srv607878581=89, srv1588504420=36, srv863800798=95, srv251907002=68, srv1099379848=9, srv247280782=67, srv397774068=77, srv2059371526=61, srv1635892467=40, srv1165663999=11, srv1416194945=30, srv1692054239=44, srv16045101=38, srv1588820842=37, srv1769629450=48, srv1336406=27, srv1334464643=26, srv1529451536=33, srv1086704446=8, srv1227514126=17, srv1324055188=23, srv1559590001=35, srv2144453205=66, srv750776119=94, srv892940418=98, srv1196961439=14, srv1070843140=6, srv1258672475=20, srv190895191=56, srv695020699=91, srv414568268=79, srv2020068472=58, srv1933124178=57, srv342301478=75, srv1124788687=10, srv2119563540=64, srv1622176542=39, srv172504275=46, srv865441434=96, srv1216505715=16, srv2059893869=62, srv1663635688=41, srv1248340678=18, srv101335444=0, srv479658305=84, srv1688997183=43, srv1847094714=50, srv1441754313=31, srv1286505947=21, srv1254857429=19, srv1868326145=52, srv1906045865=55, srv1207435503=15, srv1192211390=13, srv436710960=80, srv400924074=78, srv1544344181=34, srv1328979338=24, srv515198387=85, srv1055000512=4, srv1866176027=51, srv1064588497=5, srv1178809611=12, srv261590855=69, srv475479231=83} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-10 10:53:08,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-10 10:53:08,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-10 10:53:08,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-10 10:53:08,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-10 10:53:08,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-10 10:53:08,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-10 10:53:08,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-10 10:53:08,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-10 10:53:08,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-10 10:53:08,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-10 10:53:08,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-10 10:53:08,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-10 10:53:08,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-10 10:53:08,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-10 10:53:08,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-10 10:53:08,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-10 10:53:08,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-10 10:53:08,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-10 10:53:08,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-10 10:53:08,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-10 10:53:08,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-10 10:53:08,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-10 10:53:08,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-10 10:53:08,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-10 10:53:08,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-10 10:53:08,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-10 10:53:08,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-10 10:53:08,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-10 10:53:08,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-10 10:53:08,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-10 10:53:08,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-10 10:53:08,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-10 10:53:08,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-10 10:53:08,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-10 10:53:08,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-10 10:53:08,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-10 10:53:08,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-10 10:53:08,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-10 10:53:08,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-10 10:53:08,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-10 10:53:08,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-10 10:53:08,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-10 10:53:08,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-10 10:53:08,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-10 10:53:08,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-10 10:53:08,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-10 10:53:08,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-10 10:53:08,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-10 10:53:08,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-10 10:53:08,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-10 10:53:08,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-10 10:53:08,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-10 10:53:08,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-10 10:53:08,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-10 10:53:08,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-10 10:53:08,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-10 10:53:08,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-10 10:53:08,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-10 10:53:08,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-10 10:53:08,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-10 10:53:08,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-10 10:53:08,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-10 10:53:08,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-10 10:53:08,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-10 10:53:08,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-10 10:53:08,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-10 10:53:08,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-10 10:53:08,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-10 10:53:08,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-10 10:53:08,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-10 10:53:08,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-10 10:53:08,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-10 10:53:08,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-10 10:53:08,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-10 10:53:08,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-10 10:53:08,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-10 10:53:08,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-10 10:53:08,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-10 10:53:08,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-10 10:53:08,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-10 10:53:08,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-10 10:53:08,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-10 10:53:08,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-10 10:53:08,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-10 10:53:08,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-10 10:53:08,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-10 10:53:08,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-10 10:53:08,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-10 10:53:08,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-10 10:53:08,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-10 10:53:08,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-10 10:53:08,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-10 10:53:08,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-10 10:53:08,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-10 10:53:08,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-10 10:53:08,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-10 10:53:08,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-10 10:53:08,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-10 10:53:08,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-10 10:53:08,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-10 10:53:08,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-10 10:53:08,859 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04075145252507087, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05804329047377973, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04075145252507087, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1735, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24291666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-10 10:53:08,860 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-10 10:53:08,860 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17852472014012324, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04075145252507087, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05804329047377973, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04075145252507087, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1735, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24291666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-10 10:53:09,168 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 416 ms to try 5000 different iterations.  Found a solution that moves 2675 regions; Going from a computed imbalance of 0.17852472014012324 to a new imbalance of 0.014951457190151988. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02026941619957159); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034009967509705495, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2972222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02026941619957159); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04808333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-10 10:53:09,179 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv892940418:76 , srv400924074:79 , srv316203919:80 , srv1070843140:81 , srv1216505715:81 , srv16045101:81 , srv1664820251:81 , srv1055000512:82 , srv1064588497:82 , srv1481650629:82 , srv1544344181:82 , srv1905238773:82 , srv1933124178:82 , srv251907002:82 , srv863800798:82 , srv868227428:82 , srv1334464643:83 , srv1588820842:83 , srv1769629450:83 , srv2068079076:83 , srv2121270026:83 , srv306781200:83 , srv548587880:83 , srv740684454:83 , srv1178809611:84 , srv1324055188:84 , srv1401539833:84 , srv1816229645:84 , srv1868326145:84 , srv190895191:84 , srv2049287243:84 , srv2059371526:84 , srv2144453205:84 , srv454119414:84 , srv1014695832:85 , srv1099379848:85 , srv1192211390:85 , srv1906045865:85 , srv461548863:85 , srv582344527:85 , srv101335444:86 , srv1086704446:86 , srv1254857429:86 , srv1336406:86 , srv1416194945:86 , srv1441754313:86 , srv1529451536:86 , srv1559590001:86 , srv1635892467:86 , srv172504275:86 , srv2059893869:86 , srv342301478:86 , srv397774068:86 , srv750776119:86 , srv1043701192:87 , srv1080454172:87 , srv1124788687:87 , srv1286505947:87 , srv1328979338:87 , srv1718243420:87 , srv1847094714:87 , srv2020068472:87 , srv261590855:87 , srv303305410:87 , srv311244598:87 , srv394080521:87 , srv515198387:87 , srv72904101:87 , srv1028201144:88 , srv1227514126:88 , srv1248340678:88 , srv1330246072:88 , srv1366279572:88 , srv1663635688:88 , srv1880977814:88 , srv247280782:88 , srv281469621:88 , srv475479231:88 , srv479658305:88 , srv644535686:88 , srv953891198:88 , srv1207435503:89 , srv1258672475:89 , srv1588504420:89 , srv1745418923:89 , srv1866176027:89 , srv436710960:89 , srv1165663999:90 , srv1622176542:90 , srv1688997183:90 , srv414568268:90 , srv589040864:90 , srv695020699:90 , srv1311907108:91 , srv205372954:91 , srv2119563540:91 , srv607878581:91 , srv865441434:91 , srv1692054239:92 , srv1196961439:502 }
2024-02-10 10:53:09,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-10 10:53:09,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2068079076=63, srv953891198=99, srv1880977814=53, srv1401539833=29, srv868227428=97, srv1718243420=45, srv1745418923=47, srv2049287243=59, srv281469621=70, srv205372954=60, srv740684454=93, srv306781200=72, srv316203919=74, srv1080454172=7, srv394080521=76, srv582344527=87, srv1816229645=49, srv589040864=88, srv454119414=81, srv461548863=82, srv1330246072=25, srv2121270026=65, srv1311907108=22, srv72904101=92, srv1664820251=42, srv1014695832=1, srv548587880=86, srv303305410=71, srv1366279572=28, srv1905238773=54, srv311244598=73, srv1481650629=32, srv1028201144=2, srv1043701192=3, srv644535686=90, srv607878581=89, srv1588504420=36, srv863800798=95, srv251907002=68, srv1099379848=9, srv247280782=67, srv397774068=77, srv2059371526=61, srv1635892467=40, srv1165663999=11, srv1416194945=30, srv1692054239=44, srv16045101=38, srv1588820842=37, srv1769629450=48, srv1336406=27, srv1334464643=26, srv1529451536=33, srv1086704446=8, srv1227514126=17, srv1324055188=23, srv1559590001=35, srv2144453205=66, srv750776119=94, srv892940418=98, srv1196961439=14, srv1070843140=6, srv1258672475=20, srv190895191=56, srv695020699=91, srv414568268=79, srv2020068472=58, srv1933124178=57, srv342301478=75, srv1124788687=10, srv2119563540=64, srv1622176542=39, srv172504275=46, srv865441434=96, srv1216505715=16, srv2059893869=62, srv1663635688=41, srv1248340678=18, srv101335444=0, srv479658305=84, srv1688997183=43, srv1847094714=50, srv1441754313=31, srv1286505947=21, srv1254857429=19, srv1868326145=52, srv1906045865=55, srv1207435503=15, srv1192211390=13, srv436710960=80, srv400924074=78, srv1544344181=34, srv1328979338=24, srv515198387=85, srv1055000512=4, srv1866176027=51, srv1064588497=5, srv1178809611=12, srv261590855=69, srv475479231=83} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-10 10:53:09,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-10 10:53:09,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-10 10:53:09,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-10 10:53:09,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-10 10:53:09,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-10 10:53:09,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-10 10:53:09,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-10 10:53:09,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-10 10:53:09,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-10 10:53:09,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-10 10:53:09,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-10 10:53:09,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-10 10:53:09,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-10 10:53:09,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-10 10:53:09,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-10 10:53:09,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-10 10:53:09,223 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-10 10:53:09,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-10 10:53:09,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-10 10:53:09,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-10 10:53:09,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-10 10:53:09,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-10 10:53:09,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-10 10:53:09,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-10 10:53:09,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-10 10:53:09,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-10 10:53:09,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-10 10:53:09,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-10 10:53:09,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-10 10:53:09,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-10 10:53:09,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-10 10:53:09,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-10 10:53:09,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-10 10:53:09,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-10 10:53:09,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-10 10:53:09,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-10 10:53:09,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-10 10:53:09,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-10 10:53:09,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-10 10:53:09,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-10 10:53:09,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-10 10:53:09,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-10 10:53:09,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-10 10:53:09,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-10 10:53:09,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-10 10:53:09,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-10 10:53:09,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-10 10:53:09,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-10 10:53:09,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-10 10:53:09,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-10 10:53:09,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-10 10:53:09,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-10 10:53:09,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-10 10:53:09,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-10 10:53:09,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-10 10:53:09,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-10 10:53:09,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-10 10:53:09,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-10 10:53:09,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-10 10:53:09,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-10 10:53:09,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-10 10:53:09,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-10 10:53:09,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-10 10:53:09,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-10 10:53:09,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-10 10:53:09,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-10 10:53:09,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-10 10:53:09,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-10 10:53:09,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-10 10:53:09,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-10 10:53:09,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-10 10:53:09,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-10 10:53:09,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-10 10:53:09,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-10 10:53:09,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-10 10:53:09,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-10 10:53:09,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-10 10:53:09,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-10 10:53:09,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-10 10:53:09,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-10 10:53:09,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-10 10:53:09,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-10 10:53:09,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-10 10:53:09,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-10 10:53:09,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-10 10:53:09,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-10 10:53:09,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-10 10:53:09,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-10 10:53:09,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-10 10:53:09,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-10 10:53:09,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-10 10:53:09,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-10 10:53:09,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-10 10:53:09,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-10 10:53:09,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-10 10:53:09,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-10 10:53:09,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-10 10:53:09,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-10 10:53:09,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-10 10:53:09,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-10 10:53:09,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-10 10:53:09,374 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02026941619957159); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034009967509705495, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02026941619957159); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04808333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-10 10:53:09,375 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-10 10:53:09,375 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014932726439811634, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02026941619957159); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034009967509705495, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02026941619957159); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04808333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-10 10:53:09,691 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 439 ms to try 5000 different iterations.  Found a solution that moves 1048 regions; Going from a computed imbalance of 0.014932726439811634 to a new imbalance of 7.131382665643309E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004007724179492663); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009174698042719672); MoveCostFunction : (multiplier=7.0, imbalance=0.11644444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004007724179492663); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-10 10:53:09,693 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1165663999:82 , srv1688997183:82 , srv1014695832:83 , srv1692054239:83 , srv172504275:83 , srv1847094714:83 , srv1866176027:83 , srv1880977814:83 , srv205372954:83 , srv247280782:83 , srv281469621:83 , srv479658305:83 , srv72904101:83 , srv1207435503:84 , srv1227514126:84 , srv1330246072:84 , srv1334464643:84 , srv1401539833:84 , srv1416194945:84 , srv1559590001:84 , srv2121270026:84 , srv2144453205:84 , srv261590855:84 , srv306781200:84 , srv475479231:84 , srv607878581:84 , srv644535686:84 , srv865441434:84 , srv101335444:85 , srv1055000512:85 , srv1086704446:85 , srv1248340678:85 , srv1258672475:85 , srv1286505947:85 , srv1311907108:85 , srv1622176542:85 , srv1664820251:85 , srv1816229645:85 , srv1905238773:85 , srv2059893869:85 , srv2068079076:85 , srv2119563540:85 , srv303305410:85 , srv311244598:85 , srv316203919:85 , srv342301478:85 , srv400924074:85 , srv414568268:85 , srv695020699:85 , srv750776119:85 , srv868227428:85 , srv892940418:85 , srv1178809611:86 , srv1192211390:86 , srv1328979338:86 , srv1336406:86 , srv1441754313:86 , srv1481650629:86 , srv1529451536:86 , srv16045101:86 , srv1718243420:86 , srv1745418923:86 , srv1906045865:86 , srv190895191:86 , srv2020068472:86 , srv2049287243:86 , srv394080521:86 , srv436710960:86 , srv461548863:86 , srv582344527:86 , srv589040864:86 , srv953891198:86 , srv1043701192:87 , srv1064588497:87 , srv1070843140:87 , srv1080454172:87 , srv1324055188:87 , srv1366279572:87 , srv1588504420:87 , srv1588820842:87 , srv1635892467:87 , srv1933124178:87 , srv2059371526:87 , srv397774068:87 , srv454119414:87 , srv515198387:87 , srv548587880:87 , srv740684454:87 , srv1028201144:88 , srv1099379848:88 , srv1124788687:88 , srv1254857429:88 , srv1544344181:88 , srv1663635688:88 , srv1868326145:88 , srv251907002:88 , srv1769629450:89 , srv863800798:89 , srv1216505715:90 , srv1196961439:539 }
2024-02-10 10:53:09,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-10 10:53:09,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2068079076=63, srv953891198=99, srv1880977814=53, srv1401539833=29, srv868227428=97, srv1718243420=45, srv1745418923=47, srv2049287243=59, srv281469621=70, srv205372954=60, srv740684454=93, srv306781200=72, srv316203919=74, srv1080454172=7, srv394080521=76, srv582344527=87, srv1816229645=49, srv589040864=88, srv454119414=81, srv461548863=82, srv1330246072=25, srv2121270026=65, srv1311907108=22, srv72904101=92, srv1664820251=42, srv1014695832=1, srv548587880=86, srv303305410=71, srv1366279572=28, srv1905238773=54, srv311244598=73, srv1481650629=32, srv1028201144=2, srv1043701192=3, srv644535686=90, srv607878581=89, srv1588504420=36, srv863800798=95, srv251907002=68, srv1099379848=9, srv247280782=67, srv397774068=77, srv2059371526=61, srv1635892467=40, srv1165663999=11, srv1416194945=30, srv1692054239=44, srv16045101=38, srv1588820842=37, srv1769629450=48, srv1336406=27, srv1334464643=26, srv1529451536=33, srv1086704446=8, srv1227514126=17, srv1324055188=23, srv1559590001=35, srv2144453205=66, srv750776119=94, srv892940418=98, srv1196961439=14, srv1070843140=6, srv1258672475=20, srv190895191=56, srv695020699=91, srv414568268=79, srv2020068472=58, srv1933124178=57, srv342301478=75, srv1124788687=10, srv2119563540=64, srv1622176542=39, srv172504275=46, srv865441434=96, srv1216505715=16, srv2059893869=62, srv1663635688=41, srv1248340678=18, srv101335444=0, srv479658305=84, srv1688997183=43, srv1847094714=50, srv1441754313=31, srv1286505947=21, srv1254857429=19, srv1868326145=52, srv1906045865=55, srv1207435503=15, srv1192211390=13, srv436710960=80, srv400924074=78, srv1544344181=34, srv1328979338=24, srv515198387=85, srv1055000512=4, srv1866176027=51, srv1064588497=5, srv1178809611=12, srv261590855=69, srv475479231=83} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-10 10:53:09,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-10 10:53:09,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-10 10:53:09,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-10 10:53:09,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-10 10:53:09,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-10 10:53:09,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-10 10:53:09,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-10 10:53:09,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-10 10:53:09,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-10 10:53:09,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-10 10:53:09,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-10 10:53:09,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-10 10:53:09,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-10 10:53:09,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-10 10:53:09,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-10 10:53:09,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-10 10:53:09,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-10 10:53:09,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-10 10:53:09,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-10 10:53:09,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-10 10:53:09,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-10 10:53:09,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-10 10:53:09,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-10 10:53:09,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-10 10:53:09,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-10 10:53:09,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-10 10:53:09,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-10 10:53:09,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-10 10:53:09,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-10 10:53:09,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-10 10:53:09,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-10 10:53:09,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-10 10:53:09,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-10 10:53:09,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-10 10:53:09,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-10 10:53:09,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-10 10:53:09,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-10 10:53:09,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-10 10:53:09,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-10 10:53:09,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-10 10:53:09,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-10 10:53:09,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-10 10:53:09,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-10 10:53:09,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-10 10:53:09,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-10 10:53:09,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-10 10:53:09,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-10 10:53:09,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-10 10:53:09,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-10 10:53:09,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-10 10:53:09,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-10 10:53:09,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-10 10:53:09,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-10 10:53:09,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-10 10:53:09,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-10 10:53:09,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-10 10:53:09,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-10 10:53:09,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-10 10:53:09,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-10 10:53:09,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-10 10:53:09,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-10 10:53:09,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-10 10:53:09,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-10 10:53:09,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-10 10:53:09,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-10 10:53:09,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-10 10:53:09,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-10 10:53:09,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-10 10:53:09,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-10 10:53:09,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-10 10:53:09,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-10 10:53:09,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-10 10:53:09,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-10 10:53:09,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-10 10:53:09,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-10 10:53:09,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-10 10:53:09,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-10 10:53:09,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-10 10:53:09,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-10 10:53:09,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-10 10:53:09,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-10 10:53:09,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-10 10:53:09,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-10 10:53:09,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-10 10:53:09,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-10 10:53:09,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-10 10:53:09,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-10 10:53:09,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-10 10:53:09,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-10 10:53:09,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-10 10:53:09,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-10 10:53:09,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-10 10:53:09,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-10 10:53:09,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-10 10:53:09,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-10 10:53:09,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-10 10:53:09,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-10 10:53:09,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-10 10:53:09,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-10 10:53:09,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-10 10:53:09,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-10 10:53:09,880 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004007724179492663); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009174698042719672); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004007724179492663); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-10 10:53:09,880 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-10 10:53:09,881 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.058000137207079E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004007724179492663); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009174698042719672); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004007724179492663); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-10 10:53:10,253 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 505 ms to try 5000 different iterations.  Found a solution that moves 215 regions; Going from a computed imbalance of 7.058000137207079E-4 to a new imbalance of 1.9280018537655462E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.02388888888888889); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-10 10:53:10,254 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1028201144:83 , srv1080454172:83 , srv1216505715:83 , srv1254857429:83 , srv1588504420:83 , srv1933124178:83 , srv2119563540:83 , srv892940418:83 , srv101335444:84 , srv1043701192:84 , srv1055000512:84 , srv1064588497:84 , srv1099379848:84 , srv1124788687:84 , srv1178809611:84 , srv1207435503:84 , srv1227514126:84 , srv1286505947:84 , srv1311907108:84 , srv1324055188:84 , srv1328979338:84 , srv1330246072:84 , srv1334464643:84 , srv1366279572:84 , srv1401539833:84 , srv1441754313:84 , srv1544344181:84 , srv1588820842:84 , srv16045101:84 , srv1635892467:84 , srv1663635688:84 , srv1664820251:84 , srv1745418923:84 , srv1868326145:84 , srv1905238773:84 , srv190895191:84 , srv2020068472:84 , srv205372954:84 , srv2059371526:84 , srv2068079076:84 , srv306781200:84 , srv311244598:84 , srv316203919:84 , srv397774068:84 , srv414568268:84 , srv436710960:84 , srv454119414:84 , srv461548863:84 , srv479658305:84 , srv515198387:84 , srv582344527:84 , srv589040864:84 , srv695020699:84 , srv72904101:84 , srv740684454:84 , srv750776119:84 , srv863800798:84 , srv865441434:84 , srv1014695832:85 , srv1070843140:85 , srv1192211390:85 , srv1248340678:85 , srv1258672475:85 , srv1336406:85 , srv1416194945:85 , srv1481650629:85 , srv1529451536:85 , srv1559590001:85 , srv1622176542:85 , srv1688997183:85 , srv1692054239:85 , srv1718243420:85 , srv172504275:85 , srv1769629450:85 , srv1816229645:85 , srv1847094714:85 , srv1866176027:85 , srv1880977814:85 , srv1906045865:85 , srv2049287243:85 , srv2059893869:85 , srv2121270026:85 , srv2144453205:85 , srv247280782:85 , srv251907002:85 , srv261590855:85 , srv281469621:85 , srv303305410:85 , srv342301478:85 , srv394080521:85 , srv400924074:85 , srv475479231:85 , srv548587880:85 , srv607878581:85 , srv644535686:85 , srv868227428:85 , srv953891198:85 , srv1086704446:86 , srv1165663999:86 , srv1196961439:649 }
2024-02-10 10:53:10,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-10 10:53:10,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2068079076=63, srv953891198=99, srv1880977814=53, srv1401539833=29, srv868227428=97, srv1718243420=45, srv1745418923=47, srv2049287243=59, srv281469621=70, srv205372954=60, srv740684454=93, srv306781200=72, srv316203919=74, srv1080454172=7, srv394080521=76, srv582344527=87, srv1816229645=49, srv589040864=88, srv454119414=81, srv461548863=82, srv1330246072=25, srv2121270026=65, srv1311907108=22, srv72904101=92, srv1664820251=42, srv1014695832=1, srv548587880=86, srv303305410=71, srv1366279572=28, srv1905238773=54, srv311244598=73, srv1481650629=32, srv1028201144=2, srv1043701192=3, srv644535686=90, srv607878581=89, srv1588504420=36, srv863800798=95, srv251907002=68, srv1099379848=9, srv247280782=67, srv397774068=77, srv2059371526=61, srv1635892467=40, srv1165663999=11, srv1416194945=30, srv1692054239=44, srv16045101=38, srv1588820842=37, srv1769629450=48, srv1336406=27, srv1334464643=26, srv1529451536=33, srv1086704446=8, srv1227514126=17, srv1324055188=23, srv1559590001=35, srv2144453205=66, srv750776119=94, srv892940418=98, srv1196961439=14, srv1070843140=6, srv1258672475=20, srv190895191=56, srv695020699=91, srv414568268=79, srv2020068472=58, srv1933124178=57, srv342301478=75, srv1124788687=10, srv2119563540=64, srv1622176542=39, srv172504275=46, srv865441434=96, srv1216505715=16, srv2059893869=62, srv1663635688=41, srv1248340678=18, srv101335444=0, srv479658305=84, srv1688997183=43, srv1847094714=50, srv1441754313=31, srv1286505947=21, srv1254857429=19, srv1868326145=52, srv1906045865=55, srv1207435503=15, srv1192211390=13, srv436710960=80, srv400924074=78, srv1544344181=34, srv1328979338=24, srv515198387=85, srv1055000512=4, srv1866176027=51, srv1064588497=5, srv1178809611=12, srv261590855=69, srv475479231=83} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-10 10:53:10,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-10 10:53:10,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-10 10:53:10,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-10 10:53:10,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-10 10:53:10,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-10 10:53:10,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-10 10:53:10,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-10 10:53:10,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-10 10:53:10,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-10 10:53:10,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-10 10:53:10,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-10 10:53:10,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-10 10:53:10,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-10 10:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-10 10:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-10 10:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-10 10:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-10 10:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-10 10:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-10 10:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-10 10:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-10 10:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-10 10:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-10 10:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-10 10:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-10 10:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-10 10:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-10 10:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-10 10:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-10 10:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-10 10:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-10 10:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-10 10:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-10 10:53:10,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-10 10:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-10 10:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-10 10:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-10 10:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-10 10:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-10 10:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-10 10:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-10 10:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-10 10:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-10 10:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-10 10:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-10 10:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-10 10:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-10 10:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-10 10:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-10 10:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-10 10:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-10 10:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-10 10:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-10 10:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-10 10:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-10 10:53:10,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-10 10:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-10 10:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-10 10:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-10 10:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-10 10:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-10 10:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-10 10:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-10 10:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-10 10:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-10 10:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-10 10:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-10 10:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-10 10:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-10 10:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-10 10:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-10 10:53:10,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-10 10:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-10 10:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-10 10:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-10 10:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-10 10:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-10 10:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-10 10:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-10 10:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-10 10:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-10 10:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-10 10:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-10 10:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-10 10:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-10 10:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-10 10:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-10 10:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-10 10:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-10 10:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-10 10:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-10 10:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-10 10:53:10,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-10 10:53:10,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-10 10:53:10,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-10 10:53:10,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-10 10:53:10,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-10 10:53:10,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-10 10:53:10,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-10 10:53:10,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-10 10:53:10,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-10 10:53:10,373 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.777455636076716E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-10 10:53:10,373 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1028201144:83 , srv1080454172:83 , srv1216505715:83 , srv1254857429:83 , srv1588504420:83 , srv1933124178:83 , srv2119563540:83 , srv892940418:83 , srv101335444:84 , srv1043701192:84 , srv1055000512:84 , srv1064588497:84 , srv1099379848:84 , srv1124788687:84 , srv1178809611:84 , srv1207435503:84 , srv1227514126:84 , srv1286505947:84 , srv1311907108:84 , srv1324055188:84 , srv1328979338:84 , srv1330246072:84 , srv1334464643:84 , srv1366279572:84 , srv1401539833:84 , srv1441754313:84 , srv1544344181:84 , srv1588820842:84 , srv16045101:84 , srv1635892467:84 , srv1663635688:84 , srv1664820251:84 , srv1745418923:84 , srv1868326145:84 , srv1905238773:84 , srv190895191:84 , srv2020068472:84 , srv205372954:84 , srv2059371526:84 , srv2068079076:84 , srv306781200:84 , srv311244598:84 , srv316203919:84 , srv397774068:84 , srv414568268:84 , srv436710960:84 , srv454119414:84 , srv461548863:84 , srv479658305:84 , srv515198387:84 , srv582344527:84 , srv589040864:84 , srv695020699:84 , srv72904101:84 , srv740684454:84 , srv750776119:84 , srv863800798:84 , srv865441434:84 , srv1014695832:85 , srv1070843140:85 , srv1192211390:85 , srv1248340678:85 , srv1258672475:85 , srv1336406:85 , srv1416194945:85 , srv1481650629:85 , srv1529451536:85 , srv1559590001:85 , srv1622176542:85 , srv1688997183:85 , srv1692054239:85 , srv1718243420:85 , srv172504275:85 , srv1769629450:85 , srv1816229645:85 , srv1847094714:85 , srv1866176027:85 , srv1880977814:85 , srv1906045865:85 , srv2049287243:85 , srv2059893869:85 , srv2121270026:85 , srv2144453205:85 , srv247280782:85 , srv251907002:85 , srv261590855:85 , srv281469621:85 , srv303305410:85 , srv342301478:85 , srv394080521:85 , srv400924074:85 , srv475479231:85 , srv548587880:85 , srv607878581:85 , srv644535686:85 , srv868227428:85 , srv953891198:85 , srv1086704446:86 , srv1165663999:86 , srv1196961439:649 }
2024-02-10 10:53:10,403 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=276 (was 248) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=3943 (was 4714)
2024-02-10 10:53:10,410 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=276, ProcessCount=167, AvailableMemoryMB=3942
2024-02-10 10:53:10,411 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-02-10 10:53:10,411 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-02-10 10:53:10,424 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1013404241:84 , srv1013511602:84 , srv1084103807:84 , srv1102472907:84 , srv1127535157:84 , srv1169929462:84 , srv1199122145:84 , srv1337318915:84 , srv1359785515:84 , srv1389458751:84 , srv1524700327:84 , srv1621496319:84 , srv1712370212:84 , srv1732198725:84 , srv1771855115:84 , srv1817993582:84 , srv1860435370:84 , srv2000607805:84 , srv208859601:84 , srv220240721:84 , srv333288958:84 , srv360308832:84 , srv367731905:84 , srv380382351:84 , srv55300449:84 , srv769047599:84 , srv881810338:84 , srv895617744:84 , srv979325989:84 , srv1585831556:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-02-10 10:53:10,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-10 10:53:10,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2000607805=18, srv360308832=22, srv979325989=29, srv1013511602=1, srv1524700327=10, srv380382351=24, srv881810338=27, srv1013404241=0, srv1771855115=15, srv1359785515=8, srv1712370212=13, srv1621496319=12, srv1585831556=11, srv1169929462=5, srv220240721=20, srv1127535157=4, srv1389458751=9, srv895617744=28, srv367731905=23, srv769047599=26, srv1732198725=14, srv1084103807=2, srv1817993582=16, srv1337318915=7, srv1860435370=17, srv208859601=19, srv333288958=21, srv55300449=25, srv1199122145=6, srv1102472907=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-10 10:53:10,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-10 10:53:10,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-10 10:53:10,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-10 10:53:10,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-10 10:53:10,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-10 10:53:10,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-10 10:53:10,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-10 10:53:10,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-10 10:53:10,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-10 10:53:10,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-10 10:53:10,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-10 10:53:10,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-10 10:53:10,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-10 10:53:10,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-10 10:53:10,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-10 10:53:10,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-10 10:53:10,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-10 10:53:10,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-10 10:53:10,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-10 10:53:10,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-10 10:53:10,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-10 10:53:10,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-10 10:53:10,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-10 10:53:10,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-10 10:53:10,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-10 10:53:10,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-10 10:53:10,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-10 10:53:10,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-10 10:53:10,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-10 10:53:10,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-10 10:53:10,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-02-10 10:53:10,465 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.2168066676823028, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-10 10:53:10,465 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912876193994756, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.2168066676823028, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-02-10 10:55:10,444 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 7069368 different iterations.  Found a solution that moves 1862 regions; Going from a computed imbalance of 0.9912876193994756 to a new imbalance of 4.346000888939571E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6896296296296296, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-10 10:55:10,450 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1013404241:90 , srv1013511602:90 , srv1084103807:90 , srv1102472907:90 , srv1127535157:90 , srv1169929462:90 , srv1199122145:90 , srv1337318915:90 , srv1359785515:90 , srv1389458751:90 , srv1524700327:90 , srv1585831556:90 , srv1621496319:90 , srv1712370212:90 , srv1732198725:90 , srv1771855115:90 , srv1817993582:90 , srv1860435370:90 , srv2000607805:90 , srv208859601:90 , srv220240721:90 , srv333288958:90 , srv360308832:90 , srv367731905:90 , srv380382351:90 , srv55300449:90 , srv769047599:90 , srv881810338:90 , srv895617744:90 , srv979325989:90 }
2024-02-10 10:55:10,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-10 10:55:10,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2000607805=18, srv360308832=22, srv979325989=29, srv1013511602=1, srv1524700327=10, srv380382351=24, srv881810338=27, srv1013404241=0, srv1771855115=15, srv1359785515=8, srv1712370212=13, srv1621496319=12, srv1585831556=11, srv1169929462=5, srv220240721=20, srv1127535157=4, srv1389458751=9, srv895617744=28, srv367731905=23, srv769047599=26, srv1732198725=14, srv1084103807=2, srv1817993582=16, srv1337318915=7, srv1860435370=17, srv208859601=19, srv333288958=21, srv55300449=25, srv1199122145=6, srv1102472907=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-10 10:55:10,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-10 10:55:10,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-10 10:55:10,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-10 10:55:10,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-10 10:55:10,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-10 10:55:10,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-10 10:55:10,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-10 10:55:10,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-10 10:55:10,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-10 10:55:10,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-10 10:55:10,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-10 10:55:10,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-10 10:55:10,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-10 10:55:10,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-10 10:55:10,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-10 10:55:10,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-10 10:55:10,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-10 10:55:10,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-10 10:55:10,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-10 10:55:10,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-10 10:55:10,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-10 10:55:10,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-10 10:55:10,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-10 10:55:10,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-10 10:55:10,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-10 10:55:10,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-10 10:55:10,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-10 10:55:10,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-10 10:55:10,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-10 10:55:10,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-10 10:55:10,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-02-10 10:55:10,468 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-10 10:55:10,468 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1013404241:90 , srv1013511602:90 , srv1084103807:90 , srv1102472907:90 , srv1127535157:90 , srv1169929462:90 , srv1199122145:90 , srv1337318915:90 , srv1359785515:90 , srv1389458751:90 , srv1524700327:90 , srv1585831556:90 , srv1621496319:90 , srv1712370212:90 , srv1732198725:90 , srv1771855115:90 , srv1817993582:90 , srv1860435370:90 , srv2000607805:90 , srv208859601:90 , srv220240721:90 , srv333288958:90 , srv360308832:90 , srv367731905:90 , srv380382351:90 , srv55300449:90 , srv769047599:90 , srv881810338:90 , srv895617744:90 , srv979325989:90 }
2024-02-10 10:55:10,482 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=195 (was 276), ProcessCount=167 (was 167), AvailableMemoryMB=3538 (was 3942)