2023-11-30 03:53:16,183 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-11-30 03:53:16,505 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-11-30 03:53:16,621 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-11-30 03:53:16,683 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-30 03:53:16,690 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-30 03:53:16,703 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=270, ProcessCount=173, AvailableMemoryMB=9645
2023-11-30 03:53:16,705 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-30 03:53:16,706 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-30 03:53:16,844 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1011915035:84 , srv1014906103:84 , srv1015440215:84 , srv1024031526:84 , srv1029484921:84 , srv1062230074:84 , srv1080332333:84 , srv1106342948:84 , srv111199492:84 , srv1129835861:84 , srv1174257314:84 , srv1176273233:84 , srv1204960003:84 , srv1211966393:84 , srv1240563770:84 , srv1274930528:84 , srv1280374985:84 , srv1289227984:84 , srv1329300451:84 , srv1337800848:84 , srv1359560423:84 , srv1365870436:84 , srv1368242644:84 , srv137142460:84 , srv1382601379:84 , srv1384278779:84 , srv1398919795:84 , srv1402677488:84 , srv1420020594:84 , srv1433457753:84 , srv1449301779:84 , srv1484661209:84 , srv1553970699:84 , srv1564695186:84 , srv1597893228:84 , srv1606993973:84 , srv1623469433:84 , srv1633572921:84 , srv1655485546:84 , srv1664832625:84 , srv1670592809:84 , srv1677188160:84 , srv1684537513:84 , srv1685254849:84 , srv1704877220:84 , srv1708648233:84 , srv1740936417:84 , srv1755225118:84 , srv1757110146:84 , srv1783010696:84 , srv185737104:84 , srv1868041401:84 , srv1868874621:84 , srv1873801118:84 , srv187564120:84 , srv1895665641:84 , srv1901669915:84 , srv1920476856:84 , srv1933630252:84 , srv19789345:84 , srv1984500262:84 , srv1994520713:84 , srv2116384407:84 , srv2133034912:84 , srv213649369:84 , srv242238532:84 , srv254828411:84 , srv270460300:84 , srv313460023:84 , srv354604460:84 , srv368137348:84 , srv398262768:84 , srv44415274:84 , srv467455703:84 , srv474620840:84 , srv474810720:84 , srv480798457:84 , srv523789784:84 , srv526079226:84 , srv596100493:84 , srv59649919:84 , srv620778661:84 , srv623727200:84 , srv647990140:84 , srv679057817:84 , srv683837080:84 , srv694728471:84 , srv766079305:84 , srv796070655:84 , srv82191181:84 , srv822196606:84 , srv826054567:84 , srv836326497:84 , srv856570506:84 , srv896652463:84 , srv924567027:84 , srv947808969:84 , srv960726229:84 , srv966474626:84 , srv1829048650:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-11-30 03:53:16,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-30 03:53:16,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv826054567=92, srv1829048650=50, srv1664832625=39, srv398262768=72, srv1029484921=4, srv1240563770=14, srv354604460=70, srv822196606=91, srv1633572921=37, srv1359560423=20, srv523789784=78, srv1015440215=2, srv1994520713=62, srv924567027=96, srv647990140=84, srv1868041401=52, srv213649369=65, srv1708648233=45, srv1174257314=10, srv1564695186=33, srv254828411=67, srv526079226=79, srv242238532=66, srv1382601379=24, srv1757110146=48, srv1062230074=5, srv1553970699=32, srv1685254849=43, srv111199492=8, srv1129835861=9, srv1684537513=42, srv766079305=88, srv1176273233=11, srv1365870436=21, srv59649919=81, srv856570506=94, srv467455703=74, srv368137348=71, srv683837080=86, srv313460023=69, srv1933630252=59, srv1984500262=61, srv623727200=83, srv896652463=95, srv1740936417=46, srv1895665641=56, srv1402677488=27, srv480798457=77, srv1274930528=15, srv1106342948=7, srv1873801118=54, srv966474626=99, srv596100493=80, srv1080332333=6, srv1484661209=31, srv1623469433=36, srv1368242644=22, srv1868874621=53, srv1901669915=57, srv1280374985=16, srv1755225118=47, srv1449301779=30, srv1597893228=34, srv960726229=98, srv694728471=87, srv187564120=55, srv44415274=73, srv1920476856=58, srv1398919795=26, srv1655485546=38, srv679057817=85, srv1011915035=0, srv1289227984=17, srv1384278779=25, srv836326497=93, srv82191181=90, srv1420020594=28, srv796070655=89, srv1783010696=49, srv270460300=68, srv1329300451=18, srv1024031526=3, srv1677188160=41, srv185737104=51, srv474810720=76, srv947808969=97, srv137142460=23, srv1204960003=12, srv1211966393=13, srv2133034912=64, srv620778661=82, srv1433457753=29, srv1704877220=44, srv1014906103=1, srv1606993973=35, srv1337800848=19, srv2116384407=63, srv19789345=60, srv474620840=75, srv1670592809=40} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-30 03:53:16,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-30 03:53:16,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-30 03:53:16,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-30 03:53:16,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-30 03:53:16,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-30 03:53:16,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-30 03:53:16,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-30 03:53:16,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-30 03:53:16,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-30 03:53:16,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-30 03:53:16,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-30 03:53:16,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-30 03:53:16,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-30 03:53:16,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-30 03:53:16,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-30 03:53:16,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-30 03:53:16,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-30 03:53:16,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-30 03:53:16,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-30 03:53:16,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-30 03:53:16,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-30 03:53:16,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-30 03:53:16,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-30 03:53:16,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-30 03:53:16,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-30 03:53:16,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-30 03:53:16,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-30 03:53:16,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-30 03:53:16,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-30 03:53:16,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-30 03:53:16,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-30 03:53:16,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-30 03:53:16,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-30 03:53:16,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-30 03:53:16,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-30 03:53:16,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-30 03:53:16,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-30 03:53:16,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-30 03:53:16,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-30 03:53:16,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-30 03:53:16,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-30 03:53:16,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-30 03:53:16,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-30 03:53:16,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-30 03:53:16,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-30 03:53:16,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-30 03:53:16,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-30 03:53:16,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-30 03:53:16,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-30 03:53:16,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-30 03:53:16,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-30 03:53:16,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-30 03:53:16,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-30 03:53:16,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-30 03:53:16,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-30 03:53:16,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-30 03:53:16,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-30 03:53:16,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-30 03:53:16,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-30 03:53:16,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-30 03:53:16,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-30 03:53:16,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-30 03:53:16,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-30 03:53:16,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-30 03:53:16,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-30 03:53:16,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-30 03:53:16,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-30 03:53:16,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-30 03:53:16,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-30 03:53:16,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-30 03:53:16,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-30 03:53:16,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-30 03:53:16,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-30 03:53:16,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-30 03:53:16,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-30 03:53:16,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-30 03:53:16,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-30 03:53:16,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-30 03:53:16,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-30 03:53:16,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-30 03:53:16,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-30 03:53:16,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-30 03:53:16,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-30 03:53:16,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-30 03:53:16,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-30 03:53:16,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-30 03:53:16,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-30 03:53:16,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-30 03:53:16,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-30 03:53:16,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-30 03:53:16,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-30 03:53:16,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-30 03:53:16,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-30 03:53:16,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-30 03:53:16,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-30 03:53:16,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-30 03:53:16,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-30 03:53:16,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-30 03:53:16,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-30 03:53:16,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-30 03:53:16,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-30 03:53:17,340 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-30 03:53:17,341 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-30 03:53:17,341 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-30 03:53:17,766 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 766 ms to try 5000 different iterations.  Found a solution that moves 4629 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18773236784926925. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04354333413823291, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05837107965864837, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5143333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04354333413823291, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18275, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.25216666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-30 03:53:17,783 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1382601379:81 , srv1553970699:82 , srv1670592809:82 , srv1868874621:82 , srv313460023:82 , srv474620840:82 , srv526079226:82 , srv947808969:82 , srv1129835861:83 , srv1274930528:83 , srv1289227984:83 , srv1433457753:83 , srv1564695186:83 , srv1868041401:83 , srv270460300:83 , srv480798457:83 , srv523789784:83 , srv766079305:83 , srv966474626:83 , srv1204960003:84 , srv1240563770:84 , srv1365870436:84 , srv1402677488:84 , srv1708648233:84 , srv1755225118:84 , srv185737104:84 , srv2116384407:84 , srv474810720:84 , srv623727200:84 , srv856570506:84 , srv1080332333:85 , srv1106342948:85 , srv1174257314:85 , srv1359560423:85 , srv1420020594:85 , srv1623469433:85 , srv1677188160:85 , srv1901669915:85 , srv1984500262:85 , srv354604460:85 , srv596100493:85 , srv679057817:85 , srv826054567:85 , srv1011915035:86 , srv1024031526:86 , srv111199492:86 , srv1176273233:86 , srv1398919795:86 , srv1449301779:86 , srv1484661209:86 , srv1606993973:86 , srv1633572921:86 , srv2133034912:86 , srv254828411:86 , srv368137348:86 , srv398262768:86 , srv467455703:86 , srv82191181:86 , srv896652463:86 , srv1014906103:87 , srv1211966393:87 , srv1280374985:87 , srv1337800848:87 , srv1384278779:87 , srv1757110146:87 , srv1873801118:87 , srv1895665641:87 , srv1933630252:87 , srv1994520713:87 , srv924567027:87 , srv1029484921:88 , srv1597893228:88 , srv1664832625:88 , srv1685254849:88 , srv1704877220:88 , srv1783010696:88 , srv187564120:88 , srv242238532:88 , srv44415274:88 , srv59649919:88 , srv620778661:88 , srv647990140:88 , srv683837080:88 , srv694728471:88 , srv1368242644:89 , srv137142460:89 , srv1655485546:89 , srv1740936417:89 , srv213649369:89 , srv1015440215:90 , srv1062230074:90 , srv1920476856:90 , srv822196606:90 , srv960726229:90 , srv1684537513:91 , srv19789345:91 , srv836326497:94 , srv1329300451:95 , srv796070655:95 , srv1829048650:477 }
2023-11-30 03:53:17,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-30 03:53:17,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv826054567=92, srv1829048650=50, srv1664832625=39, srv398262768=72, srv1029484921=4, srv1240563770=14, srv354604460=70, srv822196606=91, srv1633572921=37, srv1359560423=20, srv523789784=78, srv1015440215=2, srv1994520713=62, srv924567027=96, srv647990140=84, srv1868041401=52, srv213649369=65, srv1708648233=45, srv1174257314=10, srv1564695186=33, srv254828411=67, srv526079226=79, srv242238532=66, srv1382601379=24, srv1757110146=48, srv1062230074=5, srv1553970699=32, srv1685254849=43, srv111199492=8, srv1129835861=9, srv1684537513=42, srv766079305=88, srv1176273233=11, srv1365870436=21, srv59649919=81, srv856570506=94, srv467455703=74, srv368137348=71, srv683837080=86, srv313460023=69, srv1933630252=59, srv1984500262=61, srv623727200=83, srv896652463=95, srv1740936417=46, srv1895665641=56, srv1402677488=27, srv480798457=77, srv1274930528=15, srv1106342948=7, srv1873801118=54, srv966474626=99, srv596100493=80, srv1080332333=6, srv1484661209=31, srv1623469433=36, srv1368242644=22, srv1868874621=53, srv1901669915=57, srv1280374985=16, srv1755225118=47, srv1449301779=30, srv1597893228=34, srv960726229=98, srv694728471=87, srv187564120=55, srv44415274=73, srv1920476856=58, srv1398919795=26, srv1655485546=38, srv679057817=85, srv1011915035=0, srv1289227984=17, srv1384278779=25, srv836326497=93, srv82191181=90, srv1420020594=28, srv796070655=89, srv1783010696=49, srv270460300=68, srv1329300451=18, srv1024031526=3, srv1677188160=41, srv185737104=51, srv474810720=76, srv947808969=97, srv137142460=23, srv1204960003=12, srv1211966393=13, srv2133034912=64, srv620778661=82, srv1433457753=29, srv1704877220=44, srv1014906103=1, srv1606993973=35, srv1337800848=19, srv2116384407=63, srv19789345=60, srv474620840=75, srv1670592809=40} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-30 03:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-30 03:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-30 03:53:17,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-30 03:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-30 03:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-30 03:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-30 03:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-30 03:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-30 03:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-30 03:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-30 03:53:17,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-30 03:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-30 03:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-30 03:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-30 03:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-30 03:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-30 03:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-30 03:53:17,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-30 03:53:17,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-30 03:53:17,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-30 03:53:17,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-30 03:53:17,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-30 03:53:17,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-30 03:53:17,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-30 03:53:17,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-30 03:53:17,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-30 03:53:17,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-30 03:53:17,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-30 03:53:17,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-30 03:53:17,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-30 03:53:17,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-30 03:53:17,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-30 03:53:17,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-30 03:53:17,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-30 03:53:17,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-30 03:53:17,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-30 03:53:17,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-30 03:53:17,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-30 03:53:17,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-30 03:53:17,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-30 03:53:17,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-30 03:53:17,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-30 03:53:17,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-30 03:53:17,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-30 03:53:17,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-30 03:53:17,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-30 03:53:17,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-30 03:53:17,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-30 03:53:17,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-30 03:53:17,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-30 03:53:17,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-30 03:53:17,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-30 03:53:17,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-30 03:53:17,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-30 03:53:17,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-30 03:53:17,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-30 03:53:17,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-30 03:53:17,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-30 03:53:17,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-30 03:53:17,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-30 03:53:17,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-30 03:53:17,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-30 03:53:17,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-30 03:53:17,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-30 03:53:17,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-30 03:53:17,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-30 03:53:17,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-30 03:53:17,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-30 03:53:17,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-30 03:53:17,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-30 03:53:17,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-30 03:53:17,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-30 03:53:17,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-30 03:53:17,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-30 03:53:17,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-30 03:53:17,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-30 03:53:17,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-30 03:53:17,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-30 03:53:17,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-30 03:53:17,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-30 03:53:17,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-30 03:53:17,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-30 03:53:17,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-30 03:53:17,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-30 03:53:17,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-30 03:53:17,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-30 03:53:17,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-30 03:53:17,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-30 03:53:17,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-30 03:53:17,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-30 03:53:17,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-30 03:53:17,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-30 03:53:17,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-30 03:53:17,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-30 03:53:17,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-30 03:53:17,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-30 03:53:17,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-30 03:53:17,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-30 03:53:17,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-30 03:53:17,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-30 03:53:17,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-30 03:53:18,001 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04354333413823291, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05837107965864837, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04354333413823291, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18275, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.25216666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-30 03:53:18,001 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-30 03:53:18,002 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18769995489849337, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04354333413823291, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05837107965864837, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04354333413823291, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18275, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.25216666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-30 03:53:18,260 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 405 ms to try 5000 different iterations.  Found a solution that moves 2691 regions; Going from a computed imbalance of 0.18769995489849337 to a new imbalance of 0.01498837173499494. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02620437293169281, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035835388712487606, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.299, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02620437293169281, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-30 03:53:18,280 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv796070655:75 , srv1329300451:78 , srv137142460:78 , srv836326497:78 , srv1684537513:80 , srv1920476856:80 , srv1368242644:81 , srv1740936417:81 , srv1062230074:82 , srv1933630252:82 , srv19789345:82 , srv44415274:82 , srv822196606:82 , srv960726229:82 , srv1014906103:83 , srv1484661209:83 , srv1655485546:83 , srv1873801118:83 , srv1895665641:83 , srv683837080:83 , srv924567027:83 , srv1029484921:84 , srv1080332333:84 , srv1211966393:84 , srv1398919795:84 , srv1664832625:84 , srv1704877220:84 , srv187564120:84 , srv242238532:84 , srv647990140:84 , srv679057817:84 , srv694728471:84 , srv1015440215:85 , srv111199492:85 , srv1280374985:85 , srv1384278779:85 , srv1597893228:85 , srv1606993973:85 , srv1633572921:85 , srv1783010696:85 , srv1984500262:85 , srv213649369:85 , srv254828411:85 , srv398262768:85 , srv620778661:85 , srv896652463:85 , srv1024031526:86 , srv1106342948:86 , srv1174257314:86 , srv1337800848:86 , srv1365870436:86 , srv1420020594:86 , srv1449301779:86 , srv1708648233:86 , srv1757110146:86 , srv185737104:86 , srv1901669915:86 , srv1994520713:86 , srv59649919:86 , srv826054567:86 , srv1129835861:87 , srv1359560423:87 , srv1677188160:87 , srv1685254849:87 , srv1868041401:87 , srv1868874621:87 , srv2133034912:87 , srv354604460:87 , srv368137348:87 , srv467455703:87 , srv623727200:87 , srv82191181:87 , srv1011915035:88 , srv1176273233:88 , srv1204960003:88 , srv1240563770:88 , srv1274930528:88 , srv1623469433:88 , srv474810720:88 , srv523789784:88 , srv596100493:88 , srv766079305:88 , srv947808969:88 , srv966474626:88 , srv1289227984:89 , srv1433457753:89 , srv1553970699:89 , srv1564695186:89 , srv1670592809:89 , srv270460300:89 , srv526079226:89 , srv1755225118:90 , srv313460023:90 , srv474620840:90 , srv856570506:90 , srv1402677488:91 , srv2116384407:91 , srv480798457:91 , srv1382601379:92 , srv1829048650:530 }
2023-11-30 03:53:18,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-30 03:53:18,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv826054567=92, srv1829048650=50, srv1664832625=39, srv398262768=72, srv1029484921=4, srv1240563770=14, srv354604460=70, srv822196606=91, srv1633572921=37, srv1359560423=20, srv523789784=78, srv1015440215=2, srv1994520713=62, srv924567027=96, srv647990140=84, srv1868041401=52, srv213649369=65, srv1708648233=45, srv1174257314=10, srv1564695186=33, srv254828411=67, srv526079226=79, srv242238532=66, srv1382601379=24, srv1757110146=48, srv1062230074=5, srv1553970699=32, srv1685254849=43, srv111199492=8, srv1129835861=9, srv1684537513=42, srv766079305=88, srv1176273233=11, srv1365870436=21, srv59649919=81, srv856570506=94, srv467455703=74, srv368137348=71, srv683837080=86, srv313460023=69, srv1933630252=59, srv1984500262=61, srv623727200=83, srv896652463=95, srv1740936417=46, srv1895665641=56, srv1402677488=27, srv480798457=77, srv1274930528=15, srv1106342948=7, srv1873801118=54, srv966474626=99, srv596100493=80, srv1080332333=6, srv1484661209=31, srv1623469433=36, srv1368242644=22, srv1868874621=53, srv1901669915=57, srv1280374985=16, srv1755225118=47, srv1449301779=30, srv1597893228=34, srv960726229=98, srv694728471=87, srv187564120=55, srv44415274=73, srv1920476856=58, srv1398919795=26, srv1655485546=38, srv679057817=85, srv1011915035=0, srv1289227984=17, srv1384278779=25, srv836326497=93, srv82191181=90, srv1420020594=28, srv796070655=89, srv1783010696=49, srv270460300=68, srv1329300451=18, srv1024031526=3, srv1677188160=41, srv185737104=51, srv474810720=76, srv947808969=97, srv137142460=23, srv1204960003=12, srv1211966393=13, srv2133034912=64, srv620778661=82, srv1433457753=29, srv1704877220=44, srv1014906103=1, srv1606993973=35, srv1337800848=19, srv2116384407=63, srv19789345=60, srv474620840=75, srv1670592809=40} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-30 03:53:18,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-30 03:53:18,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-30 03:53:18,310 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-30 03:53:18,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-30 03:53:18,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-30 03:53:18,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-30 03:53:18,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-30 03:53:18,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-30 03:53:18,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-30 03:53:18,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-30 03:53:18,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-30 03:53:18,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-30 03:53:18,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-30 03:53:18,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-30 03:53:18,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-30 03:53:18,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-30 03:53:18,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-30 03:53:18,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-30 03:53:18,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-30 03:53:18,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-30 03:53:18,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-30 03:53:18,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-30 03:53:18,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-30 03:53:18,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-30 03:53:18,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-30 03:53:18,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-30 03:53:18,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-30 03:53:18,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-30 03:53:18,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-30 03:53:18,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-30 03:53:18,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-30 03:53:18,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-30 03:53:18,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-30 03:53:18,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-30 03:53:18,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-30 03:53:18,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-30 03:53:18,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-30 03:53:18,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-30 03:53:18,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-30 03:53:18,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-30 03:53:18,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-30 03:53:18,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-30 03:53:18,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-30 03:53:18,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-30 03:53:18,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-30 03:53:18,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-30 03:53:18,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-30 03:53:18,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-30 03:53:18,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-30 03:53:18,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-30 03:53:18,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-30 03:53:18,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-30 03:53:18,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-30 03:53:18,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-30 03:53:18,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-30 03:53:18,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-30 03:53:18,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-30 03:53:18,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-30 03:53:18,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-30 03:53:18,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-30 03:53:18,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-30 03:53:18,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-30 03:53:18,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-30 03:53:18,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-30 03:53:18,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-30 03:53:18,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-30 03:53:18,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-30 03:53:18,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-30 03:53:18,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-30 03:53:18,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-30 03:53:18,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-30 03:53:18,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-30 03:53:18,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-30 03:53:18,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-30 03:53:18,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-30 03:53:18,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-30 03:53:18,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-30 03:53:18,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-30 03:53:18,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-30 03:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-30 03:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-30 03:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-30 03:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-30 03:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-30 03:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-30 03:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-30 03:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-30 03:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-30 03:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-30 03:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-30 03:53:18,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-30 03:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-30 03:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-30 03:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-30 03:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-30 03:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-30 03:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-30 03:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-30 03:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-30 03:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-30 03:53:18,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-30 03:53:18,487 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02620437293169281, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035835388712487606, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02620437293169281, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-30 03:53:18,487 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-30 03:53:18,487 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014969528950260026, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02620437293169281, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035835388712487606, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02620437293169281, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-30 03:53:18,799 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 459 ms to try 5000 different iterations.  Found a solution that moves 1000 regions; Going from a computed imbalance of 0.014969528950260026 to a new imbalance of 9.401254748171815E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009125635134638571); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010865982916250669); MoveCostFunction : (multiplier=7.0, imbalance=0.1111111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.009125635134638571); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-30 03:53:18,801 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1015440215:82 , srv1402677488:82 , srv1685254849:82 , srv1011915035:83 , srv1176273233:83 , srv1755225118:83 , srv2116384407:83 , srv213649369:83 , srv480798457:83 , srv59649919:83 , srv856570506:83 , srv1029484921:84 , srv1329300451:84 , srv1382601379:84 , srv1597893228:84 , srv1655485546:84 , srv1757110146:84 , srv2133034912:84 , srv467455703:84 , srv474810720:84 , srv596100493:84 , srv82191181:84 , srv1024031526:85 , srv1204960003:85 , srv1240563770:85 , srv1274930528:85 , srv1280374985:85 , srv1289227984:85 , srv1337800848:85 , srv1359560423:85 , srv1420020594:85 , srv1449301779:85 , srv1564695186:85 , srv1623469433:85 , srv1664832625:85 , srv1677188160:85 , srv1684537513:85 , srv1783010696:85 , srv19789345:85 , srv1994520713:85 , srv242238532:85 , srv254828411:85 , srv313460023:85 , srv354604460:85 , srv368137348:85 , srv474620840:85 , srv526079226:85 , srv620778661:85 , srv647990140:85 , srv683837080:85 , srv694728471:85 , srv836326497:85 , srv960726229:85 , srv1062230074:86 , srv1106342948:86 , srv1174257314:86 , srv1365870436:86 , srv1384278779:86 , srv1433457753:86 , srv1553970699:86 , srv1670592809:86 , srv1704877220:86 , srv185737104:86 , srv187564120:86 , srv1895665641:86 , srv270460300:86 , srv398262768:86 , srv623727200:86 , srv111199492:87 , srv1129835861:87 , srv1211966393:87 , srv1368242644:87 , srv1398919795:87 , srv1633572921:87 , srv1740936417:87 , srv1868041401:87 , srv1868874621:87 , srv1873801118:87 , srv1901669915:87 , srv1920476856:87 , srv1933630252:87 , srv1984500262:87 , srv44415274:87 , srv523789784:87 , srv766079305:87 , srv822196606:87 , srv826054567:87 , srv896652463:87 , srv924567027:87 , srv947808969:87 , srv1080332333:88 , srv1484661209:88 , srv1606993973:88 , srv1708648233:88 , srv679057817:88 , srv796070655:88 , srv966474626:88 , srv1014906103:89 , srv137142460:90 , srv1829048650:532 }
2023-11-30 03:53:18,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-30 03:53:18,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv826054567=92, srv1829048650=50, srv1664832625=39, srv398262768=72, srv1029484921=4, srv1240563770=14, srv354604460=70, srv822196606=91, srv1633572921=37, srv1359560423=20, srv523789784=78, srv1015440215=2, srv1994520713=62, srv924567027=96, srv647990140=84, srv1868041401=52, srv213649369=65, srv1708648233=45, srv1174257314=10, srv1564695186=33, srv254828411=67, srv526079226=79, srv242238532=66, srv1382601379=24, srv1757110146=48, srv1062230074=5, srv1553970699=32, srv1685254849=43, srv111199492=8, srv1129835861=9, srv1684537513=42, srv766079305=88, srv1176273233=11, srv1365870436=21, srv59649919=81, srv856570506=94, srv467455703=74, srv368137348=71, srv683837080=86, srv313460023=69, srv1933630252=59, srv1984500262=61, srv623727200=83, srv896652463=95, srv1740936417=46, srv1895665641=56, srv1402677488=27, srv480798457=77, srv1274930528=15, srv1106342948=7, srv1873801118=54, srv966474626=99, srv596100493=80, srv1080332333=6, srv1484661209=31, srv1623469433=36, srv1368242644=22, srv1868874621=53, srv1901669915=57, srv1280374985=16, srv1755225118=47, srv1449301779=30, srv1597893228=34, srv960726229=98, srv694728471=87, srv187564120=55, srv44415274=73, srv1920476856=58, srv1398919795=26, srv1655485546=38, srv679057817=85, srv1011915035=0, srv1289227984=17, srv1384278779=25, srv836326497=93, srv82191181=90, srv1420020594=28, srv796070655=89, srv1783010696=49, srv270460300=68, srv1329300451=18, srv1024031526=3, srv1677188160=41, srv185737104=51, srv474810720=76, srv947808969=97, srv137142460=23, srv1204960003=12, srv1211966393=13, srv2133034912=64, srv620778661=82, srv1433457753=29, srv1704877220=44, srv1014906103=1, srv1606993973=35, srv1337800848=19, srv2116384407=63, srv19789345=60, srv474620840=75, srv1670592809=40} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-30 03:53:18,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-30 03:53:18,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-30 03:53:18,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-30 03:53:18,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-30 03:53:18,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-30 03:53:18,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-30 03:53:18,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-30 03:53:18,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-30 03:53:18,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-30 03:53:18,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-30 03:53:18,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-30 03:53:18,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-30 03:53:18,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-30 03:53:18,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-30 03:53:18,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-30 03:53:18,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-30 03:53:18,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-30 03:53:18,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-30 03:53:18,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-30 03:53:18,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-30 03:53:18,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-30 03:53:18,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-30 03:53:18,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-30 03:53:18,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-30 03:53:18,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-30 03:53:18,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-30 03:53:18,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-30 03:53:18,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-30 03:53:18,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-30 03:53:18,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-30 03:53:18,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-30 03:53:18,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-30 03:53:18,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-30 03:53:18,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-30 03:53:18,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-30 03:53:18,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-30 03:53:18,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-30 03:53:18,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-30 03:53:18,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-30 03:53:18,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-30 03:53:18,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-30 03:53:18,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-30 03:53:18,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-30 03:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-30 03:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-30 03:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-30 03:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-30 03:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-30 03:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-30 03:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-30 03:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-30 03:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-30 03:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-30 03:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-30 03:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-30 03:53:18,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-30 03:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-30 03:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-30 03:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-30 03:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-30 03:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-30 03:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-30 03:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-30 03:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-30 03:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-30 03:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-30 03:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-30 03:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-30 03:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-30 03:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-30 03:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-30 03:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-30 03:53:18,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-30 03:53:18,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-30 03:53:18,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-30 03:53:18,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-30 03:53:18,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-30 03:53:18,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-30 03:53:18,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-30 03:53:18,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-30 03:53:18,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-30 03:53:18,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-30 03:53:18,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-30 03:53:18,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-30 03:53:18,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-30 03:53:18,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-30 03:53:18,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-30 03:53:18,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-30 03:53:18,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-30 03:53:18,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-30 03:53:18,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-30 03:53:18,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-30 03:53:18,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-30 03:53:18,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-30 03:53:18,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-30 03:53:18,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-30 03:53:18,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-30 03:53:18,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-30 03:53:18,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-30 03:53:18,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-30 03:53:18,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-30 03:53:18,964 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009125635134638571); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010865982916250669); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.009125635134638571); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-30 03:53:18,964 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-30 03:53:18,964 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.331233251572361E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009125635134638571); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010865982916250669); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.009125635134638571); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-30 03:53:19,465 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 619 ms to try 5000 different iterations.  Found a solution that moves 300 regions; Going from a computed imbalance of 9.331233251572361E-4 to a new imbalance of 5.665366305797348E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.03333333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-30 03:53:19,467 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1606993973:83 , srv523789784:83 , srv822196606:83 , srv966474626:83 , srv1014906103:84 , srv1062230074:84 , srv111199492:84 , srv1211966393:84 , srv1240563770:84 , srv1337800848:84 , srv1384278779:84 , srv1597893228:84 , srv1623469433:84 , srv1633572921:84 , srv1704877220:84 , srv1708648233:84 , srv1783010696:84 , srv1868041401:84 , srv187564120:84 , srv1901669915:84 , srv19789345:84 , srv1994520713:84 , srv2116384407:84 , srv270460300:84 , srv354604460:84 , srv59649919:84 , srv620778661:84 , srv766079305:84 , srv82191181:84 , srv826054567:84 , srv896652463:84 , srv1011915035:85 , srv1015440215:85 , srv1024031526:85 , srv1080332333:85 , srv1106342948:85 , srv1129835861:85 , srv1174257314:85 , srv1176273233:85 , srv1204960003:85 , srv1280374985:85 , srv1289227984:85 , srv1329300451:85 , srv1368242644:85 , srv137142460:85 , srv1382601379:85 , srv1398919795:85 , srv1402677488:85 , srv1433457753:85 , srv1449301779:85 , srv1484661209:85 , srv1553970699:85 , srv1564695186:85 , srv1670592809:85 , srv1677188160:85 , srv1685254849:85 , srv1740936417:85 , srv1755225118:85 , srv1757110146:85 , srv1873801118:85 , srv1920476856:85 , srv1984500262:85 , srv2133034912:85 , srv213649369:85 , srv242238532:85 , srv313460023:85 , srv368137348:85 , srv398262768:85 , srv44415274:85 , srv467455703:85 , srv474620840:85 , srv480798457:85 , srv596100493:85 , srv623727200:85 , srv647990140:85 , srv683837080:85 , srv694728471:85 , srv796070655:85 , srv836326497:85 , srv856570506:85 , srv924567027:85 , srv947808969:85 , srv960726229:85 , srv1029484921:86 , srv1274930528:86 , srv1359560423:86 , srv1365870436:86 , srv1420020594:86 , srv1655485546:86 , srv1664832625:86 , srv1684537513:86 , srv185737104:86 , srv1868874621:86 , srv1895665641:86 , srv1933630252:86 , srv254828411:86 , srv474810720:86 , srv526079226:86 , srv679057817:86 , srv1829048650:604 }
2023-11-30 03:53:19,473 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-30 03:53:19,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv826054567=92, srv1829048650=50, srv1664832625=39, srv398262768=72, srv1029484921=4, srv1240563770=14, srv354604460=70, srv822196606=91, srv1633572921=37, srv1359560423=20, srv523789784=78, srv1015440215=2, srv1994520713=62, srv924567027=96, srv647990140=84, srv1868041401=52, srv213649369=65, srv1708648233=45, srv1174257314=10, srv1564695186=33, srv254828411=67, srv526079226=79, srv242238532=66, srv1382601379=24, srv1757110146=48, srv1062230074=5, srv1553970699=32, srv1685254849=43, srv111199492=8, srv1129835861=9, srv1684537513=42, srv766079305=88, srv1176273233=11, srv1365870436=21, srv59649919=81, srv856570506=94, srv467455703=74, srv368137348=71, srv683837080=86, srv313460023=69, srv1933630252=59, srv1984500262=61, srv623727200=83, srv896652463=95, srv1740936417=46, srv1895665641=56, srv1402677488=27, srv480798457=77, srv1274930528=15, srv1106342948=7, srv1873801118=54, srv966474626=99, srv596100493=80, srv1080332333=6, srv1484661209=31, srv1623469433=36, srv1368242644=22, srv1868874621=53, srv1901669915=57, srv1280374985=16, srv1755225118=47, srv1449301779=30, srv1597893228=34, srv960726229=98, srv694728471=87, srv187564120=55, srv44415274=73, srv1920476856=58, srv1398919795=26, srv1655485546=38, srv679057817=85, srv1011915035=0, srv1289227984=17, srv1384278779=25, srv836326497=93, srv82191181=90, srv1420020594=28, srv796070655=89, srv1783010696=49, srv270460300=68, srv1329300451=18, srv1024031526=3, srv1677188160=41, srv185737104=51, srv474810720=76, srv947808969=97, srv137142460=23, srv1204960003=12, srv1211966393=13, srv2133034912=64, srv620778661=82, srv1433457753=29, srv1704877220=44, srv1014906103=1, srv1606993973=35, srv1337800848=19, srv2116384407=63, srv19789345=60, srv474620840=75, srv1670592809=40} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-30 03:53:19,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-30 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-30 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-30 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-30 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-30 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-30 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-30 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-30 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-30 03:53:19,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-30 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-30 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-30 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-30 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-30 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-30 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-30 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-30 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-30 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-30 03:53:19,494 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-30 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-30 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-30 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-30 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-30 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-30 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-30 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-30 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-30 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-30 03:53:19,495 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-30 03:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-30 03:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-30 03:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-30 03:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-30 03:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-30 03:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-30 03:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-30 03:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-30 03:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-30 03:53:19,496 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-30 03:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-30 03:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-30 03:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-30 03:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-30 03:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-30 03:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-30 03:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-30 03:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-30 03:53:19,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-30 03:53:19,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-30 03:53:19,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-30 03:53:19,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-30 03:53:19,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-30 03:53:19,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-30 03:53:19,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-30 03:53:19,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-30 03:53:19,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-30 03:53:19,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-30 03:53:19,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-30 03:53:19,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-30 03:53:19,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-30 03:53:19,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-30 03:53:19,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-30 03:53:19,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-30 03:53:19,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-30 03:53:19,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-30 03:53:19,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-30 03:53:19,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-30 03:53:19,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-30 03:53:19,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-30 03:53:19,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-30 03:53:19,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-30 03:53:19,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-30 03:53:19,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-30 03:53:19,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-30 03:53:19,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-30 03:53:19,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-30 03:53:19,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-30 03:53:19,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-30 03:53:19,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-30 03:53:19,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-30 03:53:19,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-30 03:53:19,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-30 03:53:19,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-30 03:53:19,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-30 03:53:19,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-30 03:53:19,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-30 03:53:19,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-30 03:53:19,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-30 03:53:19,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-30 03:53:19,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-30 03:53:19,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-30 03:53:19,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-30 03:53:19,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-30 03:53:19,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-30 03:53:19,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-30 03:53:19,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-30 03:53:19,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-30 03:53:19,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-30 03:53:19,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-30 03:53:19,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-30 03:53:19,657 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-30 03:53:19,658 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-30 03:53:19,658 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.455301815998979E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-30 03:53:19,981 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 462 ms to try 5000 different iterations.  Found a solution that moves 24 regions; Going from a computed imbalance of 5.455301815998979E-5 to a new imbalance of 6.912120645670642E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0026666666666666666); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-30 03:53:19,981 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1014906103:83 , srv1359560423:83 , srv1433457753:83 , srv1664832625:83 , srv368137348:83 , srv679057817:83 , srv796070655:83 , srv1011915035:84 , srv1015440215:84 , srv1024031526:84 , srv1029484921:84 , srv1062230074:84 , srv1080332333:84 , srv1106342948:84 , srv111199492:84 , srv1129835861:84 , srv1174257314:84 , srv1176273233:84 , srv1204960003:84 , srv1211966393:84 , srv1274930528:84 , srv1280374985:84 , srv1289227984:84 , srv1329300451:84 , srv1337800848:84 , srv1365870436:84 , srv1368242644:84 , srv137142460:84 , srv1382601379:84 , srv1384278779:84 , srv1398919795:84 , srv1402677488:84 , srv1420020594:84 , srv1449301779:84 , srv1484661209:84 , srv1553970699:84 , srv1564695186:84 , srv1606993973:84 , srv1623469433:84 , srv1633572921:84 , srv1655485546:84 , srv1670592809:84 , srv1677188160:84 , srv1684537513:84 , srv1685254849:84 , srv1704877220:84 , srv1708648233:84 , srv1740936417:84 , srv1755225118:84 , srv1757110146:84 , srv1783010696:84 , srv185737104:84 , srv1868874621:84 , srv1873801118:84 , srv187564120:84 , srv1895665641:84 , srv1901669915:84 , srv1920476856:84 , srv1933630252:84 , srv19789345:84 , srv1984500262:84 , srv1994520713:84 , srv2116384407:84 , srv2133034912:84 , srv213649369:84 , srv242238532:84 , srv254828411:84 , srv270460300:84 , srv313460023:84 , srv398262768:84 , srv44415274:84 , srv467455703:84 , srv474620840:84 , srv474810720:84 , srv480798457:84 , srv526079226:84 , srv596100493:84 , srv620778661:84 , srv623727200:84 , srv647990140:84 , srv694728471:84 , srv766079305:84 , srv822196606:84 , srv826054567:84 , srv836326497:84 , srv856570506:84 , srv896652463:84 , srv924567027:84 , srv947808969:84 , srv960726229:84 , srv966474626:84 , srv1240563770:85 , srv1597893228:85 , srv1868041401:85 , srv354604460:85 , srv523789784:85 , srv59649919:85 , srv683837080:85 , srv82191181:85 , srv1829048650:683 }
2023-11-30 03:53:19,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-30 03:53:19,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv826054567=92, srv1829048650=50, srv1664832625=39, srv398262768=72, srv1029484921=4, srv1240563770=14, srv354604460=70, srv822196606=91, srv1633572921=37, srv1359560423=20, srv523789784=78, srv1015440215=2, srv1994520713=62, srv924567027=96, srv647990140=84, srv1868041401=52, srv213649369=65, srv1708648233=45, srv1174257314=10, srv1564695186=33, srv254828411=67, srv526079226=79, srv242238532=66, srv1382601379=24, srv1757110146=48, srv1062230074=5, srv1553970699=32, srv1685254849=43, srv111199492=8, srv1129835861=9, srv1684537513=42, srv766079305=88, srv1176273233=11, srv1365870436=21, srv59649919=81, srv856570506=94, srv467455703=74, srv368137348=71, srv683837080=86, srv313460023=69, srv1933630252=59, srv1984500262=61, srv623727200=83, srv896652463=95, srv1740936417=46, srv1895665641=56, srv1402677488=27, srv480798457=77, srv1274930528=15, srv1106342948=7, srv1873801118=54, srv966474626=99, srv596100493=80, srv1080332333=6, srv1484661209=31, srv1623469433=36, srv1368242644=22, srv1868874621=53, srv1901669915=57, srv1280374985=16, srv1755225118=47, srv1449301779=30, srv1597893228=34, srv960726229=98, srv694728471=87, srv187564120=55, srv44415274=73, srv1920476856=58, srv1398919795=26, srv1655485546=38, srv679057817=85, srv1011915035=0, srv1289227984=17, srv1384278779=25, srv836326497=93, srv82191181=90, srv1420020594=28, srv796070655=89, srv1783010696=49, srv270460300=68, srv1329300451=18, srv1024031526=3, srv1677188160=41, srv185737104=51, srv474810720=76, srv947808969=97, srv137142460=23, srv1204960003=12, srv1211966393=13, srv2133034912=64, srv620778661=82, srv1433457753=29, srv1704877220=44, srv1014906103=1, srv1606993973=35, srv1337800848=19, srv2116384407=63, srv19789345=60, srv474620840=75, srv1670592809=40} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-30 03:53:20,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-30 03:53:20,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-30 03:53:20,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-30 03:53:20,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-30 03:53:20,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-30 03:53:20,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-30 03:53:20,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-30 03:53:20,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-30 03:53:20,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-30 03:53:20,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-30 03:53:20,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-30 03:53:20,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-30 03:53:20,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-30 03:53:20,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-30 03:53:20,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-30 03:53:20,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-30 03:53:20,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-30 03:53:20,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-30 03:53:20,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-30 03:53:20,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-30 03:53:20,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-30 03:53:20,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-30 03:53:20,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-30 03:53:20,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-30 03:53:20,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-30 03:53:20,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-30 03:53:20,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-30 03:53:20,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-30 03:53:20,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-30 03:53:20,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-30 03:53:20,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-30 03:53:20,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-30 03:53:20,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-30 03:53:20,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-30 03:53:20,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-30 03:53:20,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-30 03:53:20,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-30 03:53:20,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-30 03:53:20,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-30 03:53:20,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-30 03:53:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-30 03:53:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-30 03:53:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-30 03:53:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-30 03:53:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-30 03:53:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-30 03:53:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-30 03:53:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-30 03:53:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-30 03:53:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-30 03:53:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-30 03:53:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-30 03:53:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-30 03:53:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-30 03:53:20,018 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-30 03:53:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-30 03:53:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-30 03:53:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-30 03:53:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-30 03:53:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-30 03:53:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-30 03:53:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-30 03:53:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-30 03:53:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-30 03:53:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-30 03:53:20,019 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-30 03:53:20,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-30 03:53:20,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-30 03:53:20,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-30 03:53:20,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-30 03:53:20,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-30 03:53:20,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-30 03:53:20,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-30 03:53:20,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-30 03:53:20,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-30 03:53:20,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-30 03:53:20,020 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-30 03:53:20,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-30 03:53:20,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-30 03:53:20,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-30 03:53:20,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-30 03:53:20,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-30 03:53:20,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-30 03:53:20,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-30 03:53:20,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-30 03:53:20,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-30 03:53:20,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-30 03:53:20,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-30 03:53:20,021 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-30 03:53:20,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-30 03:53:20,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-30 03:53:20,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-30 03:53:20,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-30 03:53:20,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-30 03:53:20,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-30 03:53:20,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-30 03:53:20,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-30 03:53:20,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-30 03:53:20,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-30 03:53:20,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-30 03:53:20,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-30 03:53:20,134 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-30 03:53:20,134 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1014906103:83 , srv1359560423:83 , srv1433457753:83 , srv1664832625:83 , srv368137348:83 , srv679057817:83 , srv796070655:83 , srv1011915035:84 , srv1015440215:84 , srv1024031526:84 , srv1029484921:84 , srv1062230074:84 , srv1080332333:84 , srv1106342948:84 , srv111199492:84 , srv1129835861:84 , srv1174257314:84 , srv1176273233:84 , srv1204960003:84 , srv1211966393:84 , srv1274930528:84 , srv1280374985:84 , srv1289227984:84 , srv1329300451:84 , srv1337800848:84 , srv1365870436:84 , srv1368242644:84 , srv137142460:84 , srv1382601379:84 , srv1384278779:84 , srv1398919795:84 , srv1402677488:84 , srv1420020594:84 , srv1449301779:84 , srv1484661209:84 , srv1553970699:84 , srv1564695186:84 , srv1606993973:84 , srv1623469433:84 , srv1633572921:84 , srv1655485546:84 , srv1670592809:84 , srv1677188160:84 , srv1684537513:84 , srv1685254849:84 , srv1704877220:84 , srv1708648233:84 , srv1740936417:84 , srv1755225118:84 , srv1757110146:84 , srv1783010696:84 , srv185737104:84 , srv1868874621:84 , srv1873801118:84 , srv187564120:84 , srv1895665641:84 , srv1901669915:84 , srv1920476856:84 , srv1933630252:84 , srv19789345:84 , srv1984500262:84 , srv1994520713:84 , srv2116384407:84 , srv2133034912:84 , srv213649369:84 , srv242238532:84 , srv254828411:84 , srv270460300:84 , srv313460023:84 , srv398262768:84 , srv44415274:84 , srv467455703:84 , srv474620840:84 , srv474810720:84 , srv480798457:84 , srv526079226:84 , srv596100493:84 , srv620778661:84 , srv623727200:84 , srv647990140:84 , srv694728471:84 , srv766079305:84 , srv822196606:84 , srv826054567:84 , srv836326497:84 , srv856570506:84 , srv896652463:84 , srv924567027:84 , srv947808969:84 , srv960726229:84 , srv966474626:84 , srv1240563770:85 , srv1597893228:85 , srv1868041401:85 , srv354604460:85 , srv523789784:85 , srv59649919:85 , srv683837080:85 , srv82191181:85 , srv1829048650:683 }
2023-11-30 03:53:20,164 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=288 (was 270) - SystemLoadAverage LEAK? -, ProcessCount=174 (was 173) - ProcessCount LEAK? -, AvailableMemoryMB=8735 (was 9645)
2023-11-30 03:53:20,172 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=288, ProcessCount=173, AvailableMemoryMB=8735
2023-11-30 03:53:20,173 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-30 03:53:20,173 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-11-30 03:53:20,186 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1103864658:84 , srv1193549410:84 , srv1222323627:84 , srv1270532110:84 , srv128232610:84 , srv1286413665:84 , srv1306679166:84 , srv1425654858:84 , srv1528585079:84 , srv153061797:84 , srv1627667810:84 , srv1749243040:84 , srv1776777874:84 , srv1815750427:84 , srv1871686819:84 , srv2040250982:84 , srv2048361991:84 , srv2083931071:84 , srv254305062:84 , srv264735893:84 , srv267890013:84 , srv272997766:84 , srv344958815:84 , srv371042919:84 , srv472970602:84 , srv482754438:84 , srv544357879:84 , srv72046721:84 , srv93751363:84 , srv255989980:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-11-30 03:53:20,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-30 03:53:20,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv272997766=22, srv472970602=25, srv1776777874=12, srv1871686819=14, srv153061797=9, srv264735893=20, srv1286413665=5, srv1528585079=8, srv1749243040=11, srv93751363=29, srv371042919=24, srv1306679166=6, srv2083931071=17, srv1270532110=3, srv2048361991=16, srv1815750427=13, srv267890013=21, srv72046721=28, srv1425654858=7, srv482754438=26, srv1627667810=10, srv255989980=19, srv128232610=4, srv254305062=18, srv544357879=27, srv344958815=23, srv1193549410=1, srv1103864658=0, srv2040250982=15, srv1222323627=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-30 03:53:20,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-30 03:53:20,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-30 03:53:20,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-30 03:53:20,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-30 03:53:20,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-30 03:53:20,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-30 03:53:20,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-30 03:53:20,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-30 03:53:20,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-30 03:53:20,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-30 03:53:20,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-30 03:53:20,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-30 03:53:20,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-30 03:53:20,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-30 03:53:20,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-30 03:53:20,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-30 03:53:20,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-30 03:53:20,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-30 03:53:20,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-30 03:53:20,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-30 03:53:20,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-30 03:53:20,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-30 03:53:20,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-30 03:53:20,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-30 03:53:20,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-30 03:53:20,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-30 03:53:20,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-30 03:53:20,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-30 03:53:20,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-30 03:53:20,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-30 03:53:20,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-30 03:53:20,227 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1581787524526794, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-30 03:53:20,228 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912691459348245, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1581787524526794, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-11-30 03:55:20,207 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5650481 different iterations.  Found a solution that moves 1871 regions; Going from a computed imbalance of 0.9912691459348245 to a new imbalance of 4.367007337919408E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.692962962962963, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-30 03:55:20,213 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1103864658:90 , srv1193549410:90 , srv1222323627:90 , srv1270532110:90 , srv128232610:90 , srv1286413665:90 , srv1306679166:90 , srv1425654858:90 , srv1528585079:90 , srv153061797:90 , srv1627667810:90 , srv1749243040:90 , srv1776777874:90 , srv1815750427:90 , srv1871686819:90 , srv2040250982:90 , srv2048361991:90 , srv2083931071:90 , srv254305062:90 , srv255989980:90 , srv264735893:90 , srv267890013:90 , srv272997766:90 , srv344958815:90 , srv371042919:90 , srv472970602:90 , srv482754438:90 , srv544357879:90 , srv72046721:90 , srv93751363:90 }
2023-11-30 03:55:20,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-30 03:55:20,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv272997766=22, srv472970602=25, srv1776777874=12, srv1871686819=14, srv153061797=9, srv264735893=20, srv1286413665=5, srv1528585079=8, srv1749243040=11, srv93751363=29, srv371042919=24, srv1306679166=6, srv2083931071=17, srv1270532110=3, srv2048361991=16, srv1815750427=13, srv267890013=21, srv72046721=28, srv1425654858=7, srv482754438=26, srv1627667810=10, srv255989980=19, srv128232610=4, srv254305062=18, srv544357879=27, srv344958815=23, srv1193549410=1, srv1103864658=0, srv2040250982=15, srv1222323627=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-30 03:55:20,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-30 03:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-30 03:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-30 03:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-30 03:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-30 03:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-30 03:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-30 03:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-30 03:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-30 03:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-30 03:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-30 03:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-30 03:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-30 03:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-30 03:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-30 03:55:20,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-30 03:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-30 03:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-30 03:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-30 03:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-30 03:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-30 03:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-30 03:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-30 03:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-30 03:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-30 03:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-30 03:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-30 03:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-30 03:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-30 03:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-30 03:55:20,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-30 03:55:20,233 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-30 03:55:20,233 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1103864658:90 , srv1193549410:90 , srv1222323627:90 , srv1270532110:90 , srv128232610:90 , srv1286413665:90 , srv1306679166:90 , srv1425654858:90 , srv1528585079:90 , srv153061797:90 , srv1627667810:90 , srv1749243040:90 , srv1776777874:90 , srv1815750427:90 , srv1871686819:90 , srv2040250982:90 , srv2048361991:90 , srv2083931071:90 , srv254305062:90 , srv255989980:90 , srv264735893:90 , srv267890013:90 , srv272997766:90 , srv344958815:90 , srv371042919:90 , srv472970602:90 , srv482754438:90 , srv544357879:90 , srv72046721:90 , srv93751363:90 }
2023-11-30 03:55:20,247 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=184 (was 288), ProcessCount=172 (was 173), AvailableMemoryMB=8111 (was 8735)