2024-03-21 12:54:40,811 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-03-21 12:54:41,260 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-03-21 12:54:41,438 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-03-21 12:54:41,539 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-21 12:54:41,551 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-21 12:54:41,579 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=849, ProcessCount=196, AvailableMemoryMB=499
2024-03-21 12:54:41,582 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-21 12:54:41,583 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-21 12:54:41,771 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1004005424:84 , srv1030350195:84 , srv1084342102:84 , srv1084446542:84 , srv1094324121:84 , srv1097832742:84 , srv1103124983:84 , srv1106810710:84 , srv1140266217:84 , srv114935875:84 , srv1159967122:84 , srv1233079482:84 , srv1257276037:84 , srv1277354981:84 , srv1279134695:84 , srv1341621532:84 , srv1352658328:84 , srv137438198:84 , srv137869288:84 , srv1424885637:84 , srv1458792485:84 , srv1491576329:84 , srv1514187814:84 , srv1542891495:84 , srv1567807593:84 , srv1570299992:84 , srv1574868503:84 , srv1581605265:84 , srv1592841227:84 , srv1593007229:84 , srv1595973992:84 , srv1597074880:84 , srv1665898749:84 , srv1668287151:84 , srv1675335797:84 , srv168146981:84 , srv1688489975:84 , srv1698089313:84 , srv1709655960:84 , srv1739349180:84 , srv174317549:84 , srv176572763:84 , srv1797791957:84 , srv1861408737:84 , srv1865489804:84 , srv186592391:84 , srv1899828857:84 , srv1900106798:84 , srv1938021996:84 , srv1964865694:84 , srv1982550502:84 , srv1998128386:84 , srv1998753291:84 , srv2011136367:84 , srv21136212:84 , srv2139098702:84 , srv2146855358:84 , srv216396783:84 , srv250120402:84 , srv324287768:84 , srv331761715:84 , srv338503494:84 , srv345732645:84 , srv345745088:84 , srv360588166:84 , srv363600492:84 , srv37984274:84 , srv382516949:84 , srv382587177:84 , srv402529857:84 , srv43280621:84 , srv441310951:84 , srv444264309:84 , srv455788102:84 , srv512625148:84 , srv534782063:84 , srv537149869:84 , srv540022390:84 , srv552253163:84 , srv57632335:84 , srv580104102:84 , srv582009538:84 , srv650067260:84 , srv652662001:84 , srv670713217:84 , srv686208618:84 , srv696849486:84 , srv7264128:84 , srv765806450:84 , srv772505381:84 , srv788865267:84 , srv817046860:84 , srv829614395:84 , srv895234854:84 , srv916953807:84 , srv945945929:84 , srv947279228:84 , srv961656167:84 , srv97407671:84 , srv1320260794:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-03-21 12:54:41,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-21 12:54:41,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2146855358=57, srv176572763=42, srv97407671=99, srv1233079482=11, srv250120402=59, srv382587177=69, srv1279134695=14, srv1567807593=25, srv1593007229=30, srv37984274=67, srv1106810710=7, srv1982550502=51, srv817046860=92, srv1899828857=47, srv1352658328=17, srv1320260794=15, srv947279228=97, srv382516949=68, srv1140266217=8, srv1424885637=20, srv945945929=96, srv345732645=63, srv1458792485=21, srv2139098702=56, srv1542891495=24, srv186592391=46, srv345745088=64, srv788865267=91, srv686208618=86, srv1574868503=27, srv772505381=90, srv43280621=71, srv1595973992=31, srv402529857=70, srv895234854=94, srv455788102=74, srv916953807=95, srv1030350195=1, srv1277354981=13, srv1797791957=43, srv1739349180=40, srv21136212=55, srv114935875=9, srv1094324121=4, srv1675335797=35, srv1964865694=50, srv580104102=81, srv1698089313=38, srv1257276037=12, srv1709655960=39, srv1341621532=16, srv1861408737=44, srv582009538=82, srv360588166=65, srv441310951=72, srv829614395=93, srv1004005424=0, srv338503494=62, srv57632335=80, srv1097832742=5, srv324287768=60, srv216396783=58, srv363600492=66, srv1084446542=3, srv1688489975=37, srv1865489804=45, srv1581605265=28, srv1998128386=52, srv1597074880=32, srv652662001=84, srv1570299992=26, srv1491576329=22, srv670713217=85, srv2011136367=54, srv534782063=76, srv552253163=79, srv537149869=77, srv137869288=19, srv1998753291=53, srv765806450=89, srv137438198=18, srv331761715=61, srv512625148=75, srv444264309=73, srv540022390=78, srv650067260=83, srv1668287151=34, srv174317549=41, srv1514187814=23, srv1103124983=6, srv1938021996=49, srv961656167=98, srv1592841227=29, srv1159967122=10, srv1084342102=2, srv1900106798=48, srv168146981=36, srv1665898749=33, srv696849486=87, srv7264128=88} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-21 12:54:41,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-21 12:54:41,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-21 12:54:41,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-21 12:54:41,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-21 12:54:41,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-21 12:54:41,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-21 12:54:41,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-21 12:54:41,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-21 12:54:41,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-21 12:54:41,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-21 12:54:41,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-21 12:54:41,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-21 12:54:41,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-21 12:54:41,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-21 12:54:41,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-21 12:54:41,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-21 12:54:41,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-21 12:54:41,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-21 12:54:41,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-21 12:54:41,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-21 12:54:41,947 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-21 12:54:41,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-21 12:54:41,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-21 12:54:41,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-21 12:54:41,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-21 12:54:41,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-21 12:54:41,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-21 12:54:41,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-21 12:54:41,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-21 12:54:41,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-21 12:54:41,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-21 12:54:41,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-21 12:54:41,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-21 12:54:41,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-21 12:54:41,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-21 12:54:41,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-21 12:54:41,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-21 12:54:41,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-21 12:54:41,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-21 12:54:41,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-21 12:54:41,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-21 12:54:41,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-21 12:54:41,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-21 12:54:41,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-21 12:54:41,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-21 12:54:41,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-21 12:54:41,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-21 12:54:41,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-21 12:54:41,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-21 12:54:41,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-21 12:54:41,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-21 12:54:41,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-21 12:54:41,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-21 12:54:41,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-21 12:54:41,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-21 12:54:41,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-21 12:54:41,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-21 12:54:41,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-21 12:54:41,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-21 12:54:41,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-21 12:54:41,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-21 12:54:41,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-21 12:54:41,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-21 12:54:41,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-21 12:54:41,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-21 12:54:41,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-21 12:54:41,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-21 12:54:41,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-21 12:54:41,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-21 12:54:41,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-21 12:54:41,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-21 12:54:41,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-21 12:54:41,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-21 12:54:41,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-21 12:54:41,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-21 12:54:41,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-21 12:54:41,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-21 12:54:41,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-21 12:54:41,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-21 12:54:41,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-21 12:54:41,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-21 12:54:41,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-21 12:54:41,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-21 12:54:41,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-21 12:54:41,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-21 12:54:41,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-21 12:54:41,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-21 12:54:41,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-21 12:54:41,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-21 12:54:41,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-21 12:54:41,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-21 12:54:41,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-21 12:54:41,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-21 12:54:41,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-21 12:54:41,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-21 12:54:41,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-21 12:54:41,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-21 12:54:41,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-21 12:54:41,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-21 12:54:41,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-21 12:54:41,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-21 12:54:42,192 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-21 12:54:42,192 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-21 12:54:42,193 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-21 12:54:42,606 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 596 ms to try 5000 different iterations.  Found a solution that moves 4682 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17822155200701822. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03951602710993901, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053065404448383154, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5202222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03951602710993901, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17316666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24283333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-21 12:54:42,644 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1084342102:80 , srv1581605265:82 , srv382516949:82 , srv1106810710:83 , srv1257276037:83 , srv1491576329:83 , srv1938021996:83 , srv360588166:83 , srv670713217:83 , srv765806450:83 , srv1094324121:84 , srv1140266217:84 , srv1458792485:84 , srv1665898749:84 , srv1675335797:84 , srv168146981:84 , srv176572763:84 , srv216396783:84 , srv250120402:84 , srv324287768:84 , srv444264309:84 , srv455788102:84 , srv580104102:84 , srv137438198:85 , srv1688489975:85 , srv1709655960:85 , srv1964865694:85 , srv1998753291:85 , srv2011136367:85 , srv331761715:85 , srv363600492:85 , srv652662001:85 , srv772505381:85 , srv945945929:85 , srv947279228:85 , srv961656167:85 , srv97407671:85 , srv1030350195:86 , srv1097832742:86 , srv1103124983:86 , srv1233079482:86 , srv1277354981:86 , srv1279134695:86 , srv1739349180:86 , srv1861408737:86 , srv1998128386:86 , srv2146855358:86 , srv537149869:86 , srv686208618:86 , srv895234854:86 , srv1084446542:87 , srv1341621532:87 , srv1424885637:87 , srv1542891495:87 , srv1567807593:87 , srv1574868503:87 , srv1592841227:87 , srv1597074880:87 , srv1797791957:87 , srv1865489804:87 , srv1900106798:87 , srv2139098702:87 , srv338503494:87 , srv345732645:87 , srv382587177:87 , srv402529857:87 , srv441310951:87 , srv512625148:87 , srv534782063:87 , srv582009538:87 , srv696849486:87 , srv7264128:87 , srv114935875:88 , srv1352658328:88 , srv1593007229:88 , srv1668287151:88 , srv186592391:88 , srv1982550502:88 , srv345745088:88 , srv43280621:88 , srv540022390:88 , srv788865267:88 , srv817046860:88 , srv829614395:88 , srv1159967122:89 , srv137869288:89 , srv1570299992:89 , srv57632335:89 , srv1004005424:90 , srv1514187814:90 , srv1595973992:90 , srv1698089313:90 , srv174317549:90 , srv21136212:90 , srv552253163:90 , srv650067260:90 , srv916953807:91 , srv37984274:95 , srv1899828857:102 , srv1320260794:441 }
2024-03-21 12:54:42,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-21 12:54:42,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2146855358=57, srv176572763=42, srv97407671=99, srv1233079482=11, srv250120402=59, srv382587177=69, srv1279134695=14, srv1567807593=25, srv1593007229=30, srv37984274=67, srv1106810710=7, srv1982550502=51, srv817046860=92, srv1899828857=47, srv1352658328=17, srv1320260794=15, srv947279228=97, srv382516949=68, srv1140266217=8, srv1424885637=20, srv945945929=96, srv345732645=63, srv1458792485=21, srv2139098702=56, srv1542891495=24, srv186592391=46, srv345745088=64, srv788865267=91, srv686208618=86, srv1574868503=27, srv772505381=90, srv43280621=71, srv1595973992=31, srv402529857=70, srv895234854=94, srv455788102=74, srv916953807=95, srv1030350195=1, srv1277354981=13, srv1797791957=43, srv1739349180=40, srv21136212=55, srv114935875=9, srv1094324121=4, srv1675335797=35, srv1964865694=50, srv580104102=81, srv1698089313=38, srv1257276037=12, srv1709655960=39, srv1341621532=16, srv1861408737=44, srv582009538=82, srv360588166=65, srv441310951=72, srv829614395=93, srv1004005424=0, srv338503494=62, srv57632335=80, srv1097832742=5, srv324287768=60, srv216396783=58, srv363600492=66, srv1084446542=3, srv1688489975=37, srv1865489804=45, srv1581605265=28, srv1998128386=52, srv1597074880=32, srv652662001=84, srv1570299992=26, srv1491576329=22, srv670713217=85, srv2011136367=54, srv534782063=76, srv552253163=79, srv537149869=77, srv137869288=19, srv1998753291=53, srv765806450=89, srv137438198=18, srv331761715=61, srv512625148=75, srv444264309=73, srv540022390=78, srv650067260=83, srv1668287151=34, srv174317549=41, srv1514187814=23, srv1103124983=6, srv1938021996=49, srv961656167=98, srv1592841227=29, srv1159967122=10, srv1084342102=2, srv1900106798=48, srv168146981=36, srv1665898749=33, srv696849486=87, srv7264128=88} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-21 12:54:42,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-21 12:54:42,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-21 12:54:42,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-21 12:54:42,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-21 12:54:42,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-21 12:54:42,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-21 12:54:42,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-21 12:54:42,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-21 12:54:42,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-21 12:54:42,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-21 12:54:42,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-21 12:54:42,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-21 12:54:42,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-21 12:54:42,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-21 12:54:42,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-21 12:54:42,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-21 12:54:42,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-21 12:54:42,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-21 12:54:42,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-21 12:54:42,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-21 12:54:42,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-21 12:54:42,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-21 12:54:42,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-21 12:54:42,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-21 12:54:42,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-21 12:54:42,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-21 12:54:42,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-21 12:54:42,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-21 12:54:42,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-21 12:54:42,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-21 12:54:42,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-21 12:54:42,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-21 12:54:42,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-21 12:54:42,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-21 12:54:42,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-21 12:54:42,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-21 12:54:42,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-21 12:54:42,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-21 12:54:42,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-21 12:54:42,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-21 12:54:42,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-21 12:54:42,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-21 12:54:42,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-21 12:54:42,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-21 12:54:42,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-21 12:54:42,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-21 12:54:42,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-21 12:54:42,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-21 12:54:42,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-21 12:54:42,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-21 12:54:42,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-21 12:54:42,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-21 12:54:42,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-21 12:54:42,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-21 12:54:42,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-21 12:54:42,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-21 12:54:42,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-21 12:54:42,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-21 12:54:42,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-21 12:54:42,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-21 12:54:42,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-21 12:54:42,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-21 12:54:42,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-21 12:54:42,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-21 12:54:42,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-21 12:54:42,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-21 12:54:42,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-21 12:54:42,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-21 12:54:42,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-21 12:54:42,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-21 12:54:42,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-21 12:54:42,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-21 12:54:42,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-21 12:54:42,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-21 12:54:42,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-21 12:54:42,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-21 12:54:42,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-21 12:54:42,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-21 12:54:42,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-21 12:54:42,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-21 12:54:42,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-21 12:54:42,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-21 12:54:42,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-21 12:54:42,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-21 12:54:42,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-21 12:54:42,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-21 12:54:42,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-21 12:54:42,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-21 12:54:42,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-21 12:54:42,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-21 12:54:42,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-21 12:54:42,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-21 12:54:42,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-21 12:54:42,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-21 12:54:42,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-21 12:54:42,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-21 12:54:42,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-21 12:54:42,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-21 12:54:42,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-21 12:54:42,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-21 12:54:42,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-21 12:54:42,851 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03951602710993901, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053065404448383154, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03951602710993901, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17316666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24283333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-21 12:54:42,851 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-21 12:54:42,851 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1781887679423104, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03951602710993901, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053065404448383154, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03951602710993901, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17316666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24283333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-21 12:54:43,107 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 393 ms to try 5000 different iterations.  Found a solution that moves 2740 regions; Going from a computed imbalance of 0.1781887679423104 to a new imbalance of 0.014111347577110255. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020036753858227854); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.027524094128159014, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.30444444444444446, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020036753858227854); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04491666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-21 12:54:43,119 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1899828857:71 , srv37984274:77 , srv1698089313:81 , srv1514187814:82 , srv1570299992:82 , srv1595973992:82 , srv174317549:82 , srv650067260:82 , srv1004005424:83 , srv1341621532:83 , srv1352658328:83 , srv1424885637:83 , srv21136212:83 , srv552253163:83 , srv696849486:83 , srv788865267:83 , srv817046860:83 , srv1097832742:84 , srv114935875:84 , srv1159967122:84 , srv137869288:84 , srv1592841227:84 , srv1797791957:84 , srv186592391:84 , srv1982550502:84 , srv1998128386:84 , srv2139098702:84 , srv345745088:84 , srv43280621:84 , srv7264128:84 , srv916953807:84 , srv1277354981:85 , srv1542891495:85 , srv1574868503:85 , srv1668287151:85 , srv1739349180:85 , srv1998753291:85 , srv345732645:85 , srv512625148:85 , srv534782063:85 , srv582009538:85 , srv686208618:85 , srv829614395:85 , srv97407671:85 , srv1030350195:86 , srv1084446542:86 , srv1103124983:86 , srv137438198:86 , srv1567807593:86 , srv1593007229:86 , srv1597074880:86 , srv1665898749:86 , srv1688489975:86 , srv176572763:86 , srv363600492:86 , srv382587177:86 , srv402529857:86 , srv1106810710:87 , srv1233079482:87 , srv1279134695:87 , srv1675335797:87 , srv1861408737:87 , srv1865489804:87 , srv338503494:87 , srv360588166:87 , srv441310951:87 , srv537149869:87 , srv540022390:87 , srv652662001:87 , srv765806450:87 , srv895234854:87 , srv961656167:87 , srv1094324121:88 , srv1140266217:88 , srv1458792485:88 , srv168146981:88 , srv1709655960:88 , srv1900106798:88 , srv1964865694:88 , srv2011136367:88 , srv2146855358:88 , srv216396783:88 , srv250120402:88 , srv324287768:88 , srv444264309:88 , srv455788102:88 , srv57632335:88 , srv772505381:88 , srv1491576329:89 , srv1938021996:89 , srv331761715:89 , srv670713217:89 , srv945945929:89 , srv947279228:89 , srv1581605265:90 , srv580104102:90 , srv1257276037:91 , srv1084342102:92 , srv382516949:94 , srv1320260794:511 }
2024-03-21 12:54:43,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-21 12:54:43,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2146855358=57, srv176572763=42, srv97407671=99, srv1233079482=11, srv250120402=59, srv382587177=69, srv1279134695=14, srv1567807593=25, srv1593007229=30, srv37984274=67, srv1106810710=7, srv1982550502=51, srv817046860=92, srv1899828857=47, srv1352658328=17, srv1320260794=15, srv947279228=97, srv382516949=68, srv1140266217=8, srv1424885637=20, srv945945929=96, srv345732645=63, srv1458792485=21, srv2139098702=56, srv1542891495=24, srv186592391=46, srv345745088=64, srv788865267=91, srv686208618=86, srv1574868503=27, srv772505381=90, srv43280621=71, srv1595973992=31, srv402529857=70, srv895234854=94, srv455788102=74, srv916953807=95, srv1030350195=1, srv1277354981=13, srv1797791957=43, srv1739349180=40, srv21136212=55, srv114935875=9, srv1094324121=4, srv1675335797=35, srv1964865694=50, srv580104102=81, srv1698089313=38, srv1257276037=12, srv1709655960=39, srv1341621532=16, srv1861408737=44, srv582009538=82, srv360588166=65, srv441310951=72, srv829614395=93, srv1004005424=0, srv338503494=62, srv57632335=80, srv1097832742=5, srv324287768=60, srv216396783=58, srv363600492=66, srv1084446542=3, srv1688489975=37, srv1865489804=45, srv1581605265=28, srv1998128386=52, srv1597074880=32, srv652662001=84, srv1570299992=26, srv1491576329=22, srv670713217=85, srv2011136367=54, srv534782063=76, srv552253163=79, srv537149869=77, srv137869288=19, srv1998753291=53, srv765806450=89, srv137438198=18, srv331761715=61, srv512625148=75, srv444264309=73, srv540022390=78, srv650067260=83, srv1668287151=34, srv174317549=41, srv1514187814=23, srv1103124983=6, srv1938021996=49, srv961656167=98, srv1592841227=29, srv1159967122=10, srv1084342102=2, srv1900106798=48, srv168146981=36, srv1665898749=33, srv696849486=87, srv7264128=88} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-21 12:54:43,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-21 12:54:43,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-21 12:54:43,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-21 12:54:43,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-21 12:54:43,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-21 12:54:43,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-21 12:54:43,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-21 12:54:43,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-21 12:54:43,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-21 12:54:43,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-21 12:54:43,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-21 12:54:43,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-21 12:54:43,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-21 12:54:43,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-21 12:54:43,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-21 12:54:43,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-21 12:54:43,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-21 12:54:43,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-21 12:54:43,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-21 12:54:43,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-21 12:54:43,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-21 12:54:43,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-21 12:54:43,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-21 12:54:43,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-21 12:54:43,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-21 12:54:43,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-21 12:54:43,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-21 12:54:43,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-21 12:54:43,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-21 12:54:43,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-21 12:54:43,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-21 12:54:43,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-21 12:54:43,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-21 12:54:43,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-21 12:54:43,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-21 12:54:43,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-21 12:54:43,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-21 12:54:43,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-21 12:54:43,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-21 12:54:43,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-21 12:54:43,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-21 12:54:43,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-21 12:54:43,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-21 12:54:43,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-21 12:54:43,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-21 12:54:43,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-21 12:54:43,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-21 12:54:43,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-21 12:54:43,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-21 12:54:43,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-21 12:54:43,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-21 12:54:43,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-21 12:54:43,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-21 12:54:43,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-21 12:54:43,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-21 12:54:43,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-21 12:54:43,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-21 12:54:43,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-21 12:54:43,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-21 12:54:43,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-21 12:54:43,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-21 12:54:43,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-21 12:54:43,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-21 12:54:43,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-21 12:54:43,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-21 12:54:43,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-21 12:54:43,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-21 12:54:43,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-21 12:54:43,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-21 12:54:43,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-21 12:54:43,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-21 12:54:43,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-21 12:54:43,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-21 12:54:43,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-21 12:54:43,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-21 12:54:43,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-21 12:54:43,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-21 12:54:43,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-21 12:54:43,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-21 12:54:43,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-21 12:54:43,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-21 12:54:43,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-21 12:54:43,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-21 12:54:43,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-21 12:54:43,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-21 12:54:43,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-21 12:54:43,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-21 12:54:43,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-21 12:54:43,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-21 12:54:43,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-21 12:54:43,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-21 12:54:43,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-21 12:54:43,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-21 12:54:43,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-21 12:54:43,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-21 12:54:43,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-21 12:54:43,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-21 12:54:43,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-21 12:54:43,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-21 12:54:43,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-21 12:54:43,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-21 12:54:43,294 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020036753858227854); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.027524094128159014, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020036753858227854); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04491666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-21 12:54:43,294 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-21 12:54:43,294 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014092161687042006, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020036753858227854); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.027524094128159014, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020036753858227854); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04491666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-21 12:54:43,618 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 433 ms to try 5000 different iterations.  Found a solution that moves 1023 regions; Going from a computed imbalance of 0.014092161687042006 to a new imbalance of 4.153440326195501E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004893172289450491); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005443310539518174); MoveCostFunction : (multiplier=7.0, imbalance=0.11366666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004893172289450491); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0031666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-21 12:54:43,620 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv57632335:81 , srv1593007229:82 , srv1900106798:82 , srv382516949:82 , srv540022390:82 , srv1257276037:83 , srv1668287151:83 , srv338503494:83 , srv441310951:83 , srv916953807:83 , srv947279228:83 , srv1004005424:84 , srv1084446542:84 , srv1159967122:84 , srv1233079482:84 , srv1279134695:84 , srv137869288:84 , srv1865489804:84 , srv2011136367:84 , srv2146855358:84 , srv331761715:84 , srv382587177:84 , srv455788102:84 , srv580104102:84 , srv895234854:84 , srv945945929:84 , srv1084342102:85 , srv1140266217:85 , srv1514187814:85 , srv1542891495:85 , srv1567807593:85 , srv1675335797:85 , srv1709655960:85 , srv1861408737:85 , srv186592391:85 , srv1938021996:85 , srv1964865694:85 , srv1982550502:85 , srv21136212:85 , srv216396783:85 , srv250120402:85 , srv324287768:85 , srv43280621:85 , srv444264309:85 , srv537149869:85 , srv582009538:85 , srv670713217:85 , srv772505381:85 , srv829614395:85 , srv1030350195:86 , srv1103124983:86 , srv114935875:86 , srv1277354981:86 , srv1352658328:86 , srv137438198:86 , srv1458792485:86 , srv1491576329:86 , srv1574868503:86 , srv1581605265:86 , srv1595973992:86 , srv1597074880:86 , srv168146981:86 , srv1739349180:86 , srv174317549:86 , srv1797791957:86 , srv1899828857:86 , srv2139098702:86 , srv345732645:86 , srv402529857:86 , srv512625148:86 , srv534782063:86 , srv552253163:86 , srv650067260:86 , srv686208618:86 , srv788865267:86 , srv817046860:86 , srv961656167:86 , srv1094324121:87 , srv1341621532:87 , srv1592841227:87 , srv1665898749:87 , srv1688489975:87 , srv1698089313:87 , srv176572763:87 , srv1998753291:87 , srv345745088:87 , srv363600492:87 , srv37984274:87 , srv652662001:87 , srv696849486:87 , srv7264128:87 , srv765806450:87 , srv97407671:87 , srv1106810710:88 , srv1424885637:88 , srv1570299992:88 , srv1998128386:88 , srv360588166:88 , srv1097832742:89 , srv1320260794:549 }
2024-03-21 12:54:43,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-21 12:54:43,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2146855358=57, srv176572763=42, srv97407671=99, srv1233079482=11, srv250120402=59, srv382587177=69, srv1279134695=14, srv1567807593=25, srv1593007229=30, srv37984274=67, srv1106810710=7, srv1982550502=51, srv817046860=92, srv1899828857=47, srv1352658328=17, srv1320260794=15, srv947279228=97, srv382516949=68, srv1140266217=8, srv1424885637=20, srv945945929=96, srv345732645=63, srv1458792485=21, srv2139098702=56, srv1542891495=24, srv186592391=46, srv345745088=64, srv788865267=91, srv686208618=86, srv1574868503=27, srv772505381=90, srv43280621=71, srv1595973992=31, srv402529857=70, srv895234854=94, srv455788102=74, srv916953807=95, srv1030350195=1, srv1277354981=13, srv1797791957=43, srv1739349180=40, srv21136212=55, srv114935875=9, srv1094324121=4, srv1675335797=35, srv1964865694=50, srv580104102=81, srv1698089313=38, srv1257276037=12, srv1709655960=39, srv1341621532=16, srv1861408737=44, srv582009538=82, srv360588166=65, srv441310951=72, srv829614395=93, srv1004005424=0, srv338503494=62, srv57632335=80, srv1097832742=5, srv324287768=60, srv216396783=58, srv363600492=66, srv1084446542=3, srv1688489975=37, srv1865489804=45, srv1581605265=28, srv1998128386=52, srv1597074880=32, srv652662001=84, srv1570299992=26, srv1491576329=22, srv670713217=85, srv2011136367=54, srv534782063=76, srv552253163=79, srv537149869=77, srv137869288=19, srv1998753291=53, srv765806450=89, srv137438198=18, srv331761715=61, srv512625148=75, srv444264309=73, srv540022390=78, srv650067260=83, srv1668287151=34, srv174317549=41, srv1514187814=23, srv1103124983=6, srv1938021996=49, srv961656167=98, srv1592841227=29, srv1159967122=10, srv1084342102=2, srv1900106798=48, srv168146981=36, srv1665898749=33, srv696849486=87, srv7264128=88} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-21 12:54:43,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-21 12:54:43,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-21 12:54:43,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-21 12:54:43,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-21 12:54:43,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-21 12:54:43,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-21 12:54:43,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-21 12:54:43,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-21 12:54:43,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-21 12:54:43,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-21 12:54:43,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-21 12:54:43,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-21 12:54:43,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-21 12:54:43,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-21 12:54:43,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-21 12:54:43,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-21 12:54:43,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-21 12:54:43,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-21 12:54:43,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-21 12:54:43,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-21 12:54:43,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-21 12:54:43,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-21 12:54:43,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-21 12:54:43,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-21 12:54:43,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-21 12:54:43,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-21 12:54:43,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-21 12:54:43,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-21 12:54:43,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-21 12:54:43,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-21 12:54:43,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-21 12:54:43,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-21 12:54:43,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-21 12:54:43,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-21 12:54:43,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-21 12:54:43,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-21 12:54:43,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-21 12:54:43,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-21 12:54:43,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-21 12:54:43,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-21 12:54:43,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-21 12:54:43,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-21 12:54:43,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-21 12:54:43,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-21 12:54:43,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-21 12:54:43,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-21 12:54:43,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-21 12:54:43,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-21 12:54:43,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-21 12:54:43,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-21 12:54:43,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-21 12:54:43,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-21 12:54:43,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-21 12:54:43,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-21 12:54:43,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-21 12:54:43,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-21 12:54:43,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-21 12:54:43,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-21 12:54:43,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-21 12:54:43,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-21 12:54:43,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-21 12:54:43,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-21 12:54:43,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-21 12:54:43,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-21 12:54:43,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-21 12:54:43,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-21 12:54:43,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-21 12:54:43,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-21 12:54:43,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-21 12:54:43,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-21 12:54:43,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-21 12:54:43,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-21 12:54:43,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-21 12:54:43,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-21 12:54:43,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-21 12:54:43,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-21 12:54:43,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-21 12:54:43,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-21 12:54:43,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-21 12:54:43,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-21 12:54:43,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-21 12:54:43,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-21 12:54:43,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-21 12:54:43,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-21 12:54:43,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-21 12:54:43,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-21 12:54:43,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-21 12:54:43,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-21 12:54:43,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-21 12:54:43,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-21 12:54:43,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-21 12:54:43,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-21 12:54:43,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-21 12:54:43,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-21 12:54:43,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-21 12:54:43,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-21 12:54:43,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-21 12:54:43,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-21 12:54:43,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-21 12:54:43,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-21 12:54:43,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-21 12:54:43,760 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004893172289450491); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005443310539518174); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004893172289450491); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0031666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-21 12:54:43,760 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-21 12:54:43,761 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.081808335174258E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004893172289450491); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005443310539518174); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004893172289450491); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0031666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-21 12:54:44,027 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 369 ms to try 5000 different iterations.  Found a solution that moves 218 regions; Going from a computed imbalance of 4.081808335174258E-4 to a new imbalance of 1.7623309944688513E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.02422222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-21 12:54:44,028 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv552253163:82 , srv1094324121:83 , srv1097832742:83 , srv1570299992:83 , srv1597074880:83 , srv1899828857:83 , srv345745088:83 , srv37984274:83 , srv402529857:83 , srv652662001:83 , srv1030350195:84 , srv1103124983:84 , srv1106810710:84 , srv114935875:84 , srv1424885637:84 , srv1458792485:84 , srv1491576329:84 , srv1514187814:84 , srv1567807593:84 , srv1574868503:84 , srv1581605265:84 , srv1592841227:84 , srv1595973992:84 , srv168146981:84 , srv1688489975:84 , srv1698089313:84 , srv1709655960:84 , srv174317549:84 , srv1861408737:84 , srv1865489804:84 , srv1938021996:84 , srv1964865694:84 , srv1998128386:84 , srv21136212:84 , srv2146855358:84 , srv331761715:84 , srv345732645:84 , srv360588166:84 , srv363600492:84 , srv382516949:84 , srv512625148:84 , srv534782063:84 , srv537149869:84 , srv540022390:84 , srv57632335:84 , srv580104102:84 , srv650067260:84 , srv7264128:84 , srv829614395:84 , srv945945929:84 , srv961656167:84 , srv1004005424:85 , srv1084342102:85 , srv1084446542:85 , srv1140266217:85 , srv1159967122:85 , srv1233079482:85 , srv1257276037:85 , srv1277354981:85 , srv1279134695:85 , srv1341621532:85 , srv1352658328:85 , srv137438198:85 , srv137869288:85 , srv1542891495:85 , srv1665898749:85 , srv1739349180:85 , srv176572763:85 , srv1797791957:85 , srv186592391:85 , srv1900106798:85 , srv1982550502:85 , srv1998753291:85 , srv2011136367:85 , srv2139098702:85 , srv216396783:85 , srv250120402:85 , srv324287768:85 , srv338503494:85 , srv43280621:85 , srv441310951:85 , srv444264309:85 , srv582009538:85 , srv670713217:85 , srv686208618:85 , srv696849486:85 , srv765806450:85 , srv772505381:85 , srv788865267:85 , srv817046860:85 , srv895234854:85 , srv916953807:85 , srv947279228:85 , srv97407671:85 , srv1593007229:86 , srv1668287151:86 , srv1675335797:86 , srv382587177:86 , srv455788102:86 , srv1320260794:642 }
2024-03-21 12:54:44,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-21 12:54:44,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2146855358=57, srv176572763=42, srv97407671=99, srv1233079482=11, srv250120402=59, srv382587177=69, srv1279134695=14, srv1567807593=25, srv1593007229=30, srv37984274=67, srv1106810710=7, srv1982550502=51, srv817046860=92, srv1899828857=47, srv1352658328=17, srv1320260794=15, srv947279228=97, srv382516949=68, srv1140266217=8, srv1424885637=20, srv945945929=96, srv345732645=63, srv1458792485=21, srv2139098702=56, srv1542891495=24, srv186592391=46, srv345745088=64, srv788865267=91, srv686208618=86, srv1574868503=27, srv772505381=90, srv43280621=71, srv1595973992=31, srv402529857=70, srv895234854=94, srv455788102=74, srv916953807=95, srv1030350195=1, srv1277354981=13, srv1797791957=43, srv1739349180=40, srv21136212=55, srv114935875=9, srv1094324121=4, srv1675335797=35, srv1964865694=50, srv580104102=81, srv1698089313=38, srv1257276037=12, srv1709655960=39, srv1341621532=16, srv1861408737=44, srv582009538=82, srv360588166=65, srv441310951=72, srv829614395=93, srv1004005424=0, srv338503494=62, srv57632335=80, srv1097832742=5, srv324287768=60, srv216396783=58, srv363600492=66, srv1084446542=3, srv1688489975=37, srv1865489804=45, srv1581605265=28, srv1998128386=52, srv1597074880=32, srv652662001=84, srv1570299992=26, srv1491576329=22, srv670713217=85, srv2011136367=54, srv534782063=76, srv552253163=79, srv537149869=77, srv137869288=19, srv1998753291=53, srv765806450=89, srv137438198=18, srv331761715=61, srv512625148=75, srv444264309=73, srv540022390=78, srv650067260=83, srv1668287151=34, srv174317549=41, srv1514187814=23, srv1103124983=6, srv1938021996=49, srv961656167=98, srv1592841227=29, srv1159967122=10, srv1084342102=2, srv1900106798=48, srv168146981=36, srv1665898749=33, srv696849486=87, srv7264128=88} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-21 12:54:44,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-21 12:54:44,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-21 12:54:44,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-21 12:54:44,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-21 12:54:44,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-21 12:54:44,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-21 12:54:44,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-21 12:54:44,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-21 12:54:44,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-21 12:54:44,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-21 12:54:44,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-21 12:54:44,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-21 12:54:44,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-21 12:54:44,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-21 12:54:44,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-21 12:54:44,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-21 12:54:44,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-21 12:54:44,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-21 12:54:44,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-21 12:54:44,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-21 12:54:44,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-21 12:54:44,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-21 12:54:44,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-21 12:54:44,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-21 12:54:44,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-21 12:54:44,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-21 12:54:44,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-21 12:54:44,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-21 12:54:44,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-21 12:54:44,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-21 12:54:44,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-21 12:54:44,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-21 12:54:44,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-21 12:54:44,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-21 12:54:44,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-21 12:54:44,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-21 12:54:44,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-21 12:54:44,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-21 12:54:44,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-21 12:54:44,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-21 12:54:44,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-21 12:54:44,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-21 12:54:44,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-21 12:54:44,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-21 12:54:44,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-21 12:54:44,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-21 12:54:44,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-21 12:54:44,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-21 12:54:44,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-21 12:54:44,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-21 12:54:44,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-21 12:54:44,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-21 12:54:44,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-21 12:54:44,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-21 12:54:44,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-21 12:54:44,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-21 12:54:44,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-21 12:54:44,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-21 12:54:44,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-21 12:54:44,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-21 12:54:44,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-21 12:54:44,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-21 12:54:44,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-21 12:54:44,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-21 12:54:44,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-21 12:54:44,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-21 12:54:44,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-21 12:54:44,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-21 12:54:44,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-21 12:54:44,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-21 12:54:44,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-21 12:54:44,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-21 12:54:44,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-21 12:54:44,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-21 12:54:44,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-21 12:54:44,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-21 12:54:44,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-21 12:54:44,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-21 12:54:44,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-21 12:54:44,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-21 12:54:44,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-21 12:54:44,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-21 12:54:44,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-21 12:54:44,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-21 12:54:44,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-21 12:54:44,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-21 12:54:44,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-21 12:54:44,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-21 12:54:44,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-21 12:54:44,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-21 12:54:44,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-21 12:54:44,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-21 12:54:44,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-21 12:54:44,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-21 12:54:44,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-21 12:54:44,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-21 12:54:44,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-21 12:54:44,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-21 12:54:44,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-21 12:54:44,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-21 12:54:44,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-21 12:54:44,174 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.6096841318820373E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-21 12:54:44,174 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv552253163:82 , srv1094324121:83 , srv1097832742:83 , srv1570299992:83 , srv1597074880:83 , srv1899828857:83 , srv345745088:83 , srv37984274:83 , srv402529857:83 , srv652662001:83 , srv1030350195:84 , srv1103124983:84 , srv1106810710:84 , srv114935875:84 , srv1424885637:84 , srv1458792485:84 , srv1491576329:84 , srv1514187814:84 , srv1567807593:84 , srv1574868503:84 , srv1581605265:84 , srv1592841227:84 , srv1595973992:84 , srv168146981:84 , srv1688489975:84 , srv1698089313:84 , srv1709655960:84 , srv174317549:84 , srv1861408737:84 , srv1865489804:84 , srv1938021996:84 , srv1964865694:84 , srv1998128386:84 , srv21136212:84 , srv2146855358:84 , srv331761715:84 , srv345732645:84 , srv360588166:84 , srv363600492:84 , srv382516949:84 , srv512625148:84 , srv534782063:84 , srv537149869:84 , srv540022390:84 , srv57632335:84 , srv580104102:84 , srv650067260:84 , srv7264128:84 , srv829614395:84 , srv945945929:84 , srv961656167:84 , srv1004005424:85 , srv1084342102:85 , srv1084446542:85 , srv1140266217:85 , srv1159967122:85 , srv1233079482:85 , srv1257276037:85 , srv1277354981:85 , srv1279134695:85 , srv1341621532:85 , srv1352658328:85 , srv137438198:85 , srv137869288:85 , srv1542891495:85 , srv1665898749:85 , srv1739349180:85 , srv176572763:85 , srv1797791957:85 , srv186592391:85 , srv1900106798:85 , srv1982550502:85 , srv1998753291:85 , srv2011136367:85 , srv2139098702:85 , srv216396783:85 , srv250120402:85 , srv324287768:85 , srv338503494:85 , srv43280621:85 , srv441310951:85 , srv444264309:85 , srv582009538:85 , srv670713217:85 , srv686208618:85 , srv696849486:85 , srv765806450:85 , srv772505381:85 , srv788865267:85 , srv817046860:85 , srv895234854:85 , srv916953807:85 , srv947279228:85 , srv97407671:85 , srv1593007229:86 , srv1668287151:86 , srv1675335797:86 , srv382587177:86 , srv455788102:86 , srv1320260794:642 }
2024-03-21 12:54:44,220 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=886 (was 849) - SystemLoadAverage LEAK? -, ProcessCount=195 (was 196), AvailableMemoryMB=235 (was 499)
2024-03-21 12:54:44,239 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=886, ProcessCount=195, AvailableMemoryMB=236
2024-03-21 12:54:44,240 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-21 12:54:44,240 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-03-21 12:54:44,253 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1212828049:84 , srv1235621563:84 , srv1311323335:84 , srv1493255128:84 , srv1528814764:84 , srv1586597367:84 , srv1605666773:84 , srv1613074464:84 , srv1695528129:84 , srv1765760310:84 , srv1795915832:84 , srv1800581423:84 , srv1807026305:84 , srv1964560450:84 , srv2022659205:84 , srv208130961:84 , srv228919785:84 , srv311263305:84 , srv355239070:84 , srv363002817:84 , srv434754469:84 , srv461624771:84 , srv462856455:84 , srv473240295:84 , srv715013172:84 , srv743737127:84 , srv771279421:84 , srv772089476:84 , srv788617081:84 , srv276945111:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-03-21 12:54:44,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-21 12:54:44,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1586597367=5, srv771279421=27, srv1605666773=6, srv461624771=22, srv462856455=23, srv1235621563=1, srv1311323335=2, srv355239070=19, srv311263305=18, srv743737127=26, srv1800581423=11, srv228919785=16, srv1795915832=10, srv1528814764=4, srv276945111=17, srv715013172=25, srv1613074464=7, srv772089476=28, srv1765760310=9, srv363002817=20, srv1964560450=13, srv1695528129=8, srv788617081=29, srv473240295=24, srv1807026305=12, srv1493255128=3, srv1212828049=0, srv434754469=21, srv208130961=15, srv2022659205=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-21 12:54:44,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-21 12:54:44,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-21 12:54:44,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-21 12:54:44,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-21 12:54:44,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-21 12:54:44,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-21 12:54:44,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-21 12:54:44,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-21 12:54:44,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-21 12:54:44,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-21 12:54:44,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-21 12:54:44,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-21 12:54:44,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-21 12:54:44,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-21 12:54:44,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-21 12:54:44,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-21 12:54:44,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-21 12:54:44,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-21 12:54:44,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-21 12:54:44,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-21 12:54:44,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-21 12:54:44,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-21 12:54:44,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-21 12:54:44,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-21 12:54:44,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-21 12:54:44,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-21 12:54:44,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-21 12:54:44,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-21 12:54:44,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-21 12:54:44,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-21 12:54:44,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-21 12:54:44,292 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0903850426283228, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-21 12:54:44,292 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912477843582259, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0903850426283228, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-03-21 12:56:44,273 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6936232 different iterations.  Found a solution that moves 1843 regions; Going from a computed imbalance of 0.9912477843582259 to a new imbalance of 4.3016539410932495E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6825925925925926, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-21 12:56:44,279 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1212828049:90 , srv1235621563:90 , srv1311323335:90 , srv1493255128:90 , srv1528814764:90 , srv1586597367:90 , srv1605666773:90 , srv1613074464:90 , srv1695528129:90 , srv1765760310:90 , srv1795915832:90 , srv1800581423:90 , srv1807026305:90 , srv1964560450:90 , srv2022659205:90 , srv208130961:90 , srv228919785:90 , srv276945111:90 , srv311263305:90 , srv355239070:90 , srv363002817:90 , srv434754469:90 , srv461624771:90 , srv462856455:90 , srv473240295:90 , srv715013172:90 , srv743737127:90 , srv771279421:90 , srv772089476:90 , srv788617081:90 }
2024-03-21 12:56:44,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-21 12:56:44,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1586597367=5, srv771279421=27, srv1605666773=6, srv461624771=22, srv462856455=23, srv1235621563=1, srv1311323335=2, srv355239070=19, srv311263305=18, srv743737127=26, srv1800581423=11, srv228919785=16, srv1795915832=10, srv1528814764=4, srv276945111=17, srv715013172=25, srv1613074464=7, srv772089476=28, srv1765760310=9, srv363002817=20, srv1964560450=13, srv1695528129=8, srv788617081=29, srv473240295=24, srv1807026305=12, srv1493255128=3, srv1212828049=0, srv434754469=21, srv208130961=15, srv2022659205=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-21 12:56:44,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-21 12:56:44,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-21 12:56:44,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-21 12:56:44,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-21 12:56:44,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-21 12:56:44,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-21 12:56:44,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-21 12:56:44,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-21 12:56:44,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-21 12:56:44,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-21 12:56:44,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-21 12:56:44,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-21 12:56:44,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-21 12:56:44,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-21 12:56:44,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-21 12:56:44,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-21 12:56:44,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-21 12:56:44,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-21 12:56:44,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-21 12:56:44,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-21 12:56:44,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-21 12:56:44,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-21 12:56:44,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-21 12:56:44,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-21 12:56:44,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-21 12:56:44,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-21 12:56:44,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-21 12:56:44,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-21 12:56:44,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-21 12:56:44,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-21 12:56:44,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-21 12:56:44,299 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-21 12:56:44,300 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1212828049:90 , srv1235621563:90 , srv1311323335:90 , srv1493255128:90 , srv1528814764:90 , srv1586597367:90 , srv1605666773:90 , srv1613074464:90 , srv1695528129:90 , srv1765760310:90 , srv1795915832:90 , srv1800581423:90 , srv1807026305:90 , srv1964560450:90 , srv2022659205:90 , srv208130961:90 , srv228919785:90 , srv276945111:90 , srv311263305:90 , srv355239070:90 , srv363002817:90 , srv434754469:90 , srv461624771:90 , srv462856455:90 , srv473240295:90 , srv715013172:90 , srv743737127:90 , srv771279421:90 , srv772089476:90 , srv788617081:90 }
2024-03-21 12:56:44,321 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=832 (was 886), ProcessCount=192 (was 195), AvailableMemoryMB=1734 (was 236) - AvailableMemoryMB LEAK? -