2023-03-22 02:59:09,239 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-03-22 02:59:09,563 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-03-22 02:59:09,661 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-03-22 02:59:09,718 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-22 02:59:09,724 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-22 02:59:09,734 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=458, ProcessCount=167, AvailableMemoryMB=9968
2023-03-22 02:59:09,735 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-22 02:59:09,736 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-22 02:59:09,840 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv102449838:84 , srv1035545484:84 , srv1082290447:84 , srv110523231:84 , srv1109635225:84 , srv1110330537:84 , srv1121509924:84 , srv1128205631:84 , srv115318733:84 , srv118070360:84 , srv1188998347:84 , srv1190716079:84 , srv1300857548:84 , srv1360303677:84 , srv1361423653:84 , srv1367802558:84 , srv1369039534:84 , srv1376891976:84 , srv1380490213:84 , srv1439721216:84 , srv1443575895:84 , srv1448856854:84 , srv1467253354:84 , srv14691775:84 , srv1509004861:84 , srv1510573464:84 , srv1538970341:84 , srv1545984984:84 , srv1552584077:84 , srv1569002140:84 , srv1569727056:84 , srv1597494809:84 , srv1713889842:84 , srv1721830519:84 , srv1739144215:84 , srv1743558269:84 , srv1744560888:84 , srv1777497230:84 , srv1778533678:84 , srv1790273370:84 , srv1816080225:84 , srv1818525951:84 , srv1851652324:84 , srv1853690378:84 , srv1873982448:84 , srv1892617476:84 , srv1894701879:84 , srv1900286260:84 , srv1911938:84 , srv1936090169:84 , srv1954317910:84 , srv1958123687:84 , srv197346775:84 , srv1973996813:84 , srv2039624748:84 , srv204394430:84 , srv204550218:84 , srv2066664525:84 , srv2068341176:84 , srv2133075822:84 , srv245635075:84 , srv26084775:84 , srv270211223:84 , srv270373273:84 , srv278576726:84 , srv284581471:84 , srv289955177:84 , srv290320267:84 , srv311284573:84 , srv396147642:84 , srv401210796:84 , srv414228734:84 , srv420592137:84 , srv437227166:84 , srv455254178:84 , srv477899241:84 , srv485065905:84 , srv492242005:84 , srv538598167:84 , srv545308652:84 , srv548168375:84 , srv556143086:84 , srv589766443:84 , srv600512498:84 , srv611467337:84 , srv613684367:84 , srv615772185:84 , srv620034598:84 , srv626076108:84 , srv639633096:84 , srv676587961:84 , srv746192214:84 , srv807853158:84 , srv812115261:84 , srv816538161:84 , srv822205035:84 , srv889727896:84 , srv910038803:84 , srv963008430:84 , srv698306428:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-03-22 02:59:09,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-22 02:59:09,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1035545484=1, srv545308652=79, srv1110330537=5, srv1713889842=32, srv1873982448=44, srv284581471=65, srv115318733=8, srv600512498=83, srv1082290447=2, srv548168375=80, srv2133075822=59, srv1121509924=6, srv1892617476=45, srv1958123687=51, srv290320267=67, srv1744560888=36, srv204550218=56, srv620034598=87, srv1367802558=15, srv615772185=86, srv110523231=3, srv1448856854=21, srv613684367=85, srv1851652324=42, srv1894701879=46, srv437227166=73, srv485065905=76, srv401210796=70, srv812115261=94, srv1743558269=35, srv1376891976=17, srv1777497230=37, srv311284573=68, srv420592137=72, srv910038803=98, srv1190716079=11, srv289955177=66, srv1911938=48, srv807853158=93, srv1509004861=24, srv1569727056=30, srv589766443=82, srv1721830519=33, srv639633096=89, srv698306428=91, srv1109635225=4, srv626076108=88, srv538598167=78, srv1360303677=13, srv1552584077=28, srv204394430=55, srv26084775=61, srv1369039534=16, srv1954317910=50, srv676587961=90, srv245635075=60, srv1545984984=27, srv278576726=64, srv1439721216=19, srv1900286260=47, srv1818525951=41, srv1380490213=18, srv118070360=9, srv414228734=71, srv270211223=62, srv14691775=23, srv1778533678=38, srv102449838=0, srv270373273=63, srv455254178=74, srv1188998347=10, srv822205035=96, srv1597494809=31, srv1853690378=43, srv1361423653=14, srv2066664525=57, srv1300857548=12, srv1936090169=49, srv1816080225=40, srv1443575895=20, srv963008430=99, srv492242005=77, srv1467253354=22, srv889727896=97, srv1510573464=25, srv1790273370=39, srv396147642=69, srv1973996813=53, srv197346775=52, srv556143086=81, srv1569002140=29, srv1739144215=34, srv477899241=75, srv816538161=95, srv1538970341=26, srv611467337=84, srv1128205631=7, srv2068341176=58, srv2039624748=54, srv746192214=92} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-22 02:59:09,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-22 02:59:09,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-22 02:59:09,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-22 02:59:09,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-22 02:59:09,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-22 02:59:09,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-22 02:59:09,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-22 02:59:09,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-22 02:59:09,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-22 02:59:09,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-22 02:59:09,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-22 02:59:09,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-22 02:59:09,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-22 02:59:09,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-22 02:59:09,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-22 02:59:09,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-22 02:59:09,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-22 02:59:09,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-22 02:59:09,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-22 02:59:09,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-22 02:59:09,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-22 02:59:09,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-22 02:59:09,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-22 02:59:09,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-22 02:59:09,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-22 02:59:09,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-22 02:59:09,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-22 02:59:09,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-22 02:59:09,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-22 02:59:09,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-22 02:59:09,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-22 02:59:09,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-22 02:59:09,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-22 02:59:09,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-22 02:59:09,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-22 02:59:09,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-22 02:59:09,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-22 02:59:09,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-22 02:59:09,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-22 02:59:09,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-22 02:59:09,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-22 02:59:09,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-22 02:59:09,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-22 02:59:09,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-22 02:59:09,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-22 02:59:09,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-22 02:59:09,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-22 02:59:09,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-22 02:59:09,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-22 02:59:09,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-22 02:59:09,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-22 02:59:09,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-22 02:59:09,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-22 02:59:09,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-22 02:59:09,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-22 02:59:09,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-22 02:59:09,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-22 02:59:09,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-22 02:59:09,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-22 02:59:09,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-22 02:59:09,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-22 02:59:09,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-22 02:59:09,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-22 02:59:09,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-22 02:59:09,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-22 02:59:09,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-22 02:59:09,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-22 02:59:09,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-22 02:59:09,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-22 02:59:09,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-22 02:59:09,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-22 02:59:09,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-22 02:59:09,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-22 02:59:09,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-22 02:59:09,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-22 02:59:09,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-22 02:59:09,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-22 02:59:09,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-22 02:59:09,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-22 02:59:09,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-22 02:59:09,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-22 02:59:09,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-22 02:59:09,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-22 02:59:09,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-22 02:59:09,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-22 02:59:09,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-22 02:59:09,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-22 02:59:09,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-22 02:59:09,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-22 02:59:09,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-22 02:59:09,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-22 02:59:09,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-22 02:59:09,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-22 02:59:09,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-22 02:59:09,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-22 02:59:09,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-22 02:59:09,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-22 02:59:09,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-22 02:59:09,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-22 02:59:09,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-22 02:59:09,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-22 02:59:10,127 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 02:59:10,127 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-22 02:59:10,127 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-22 02:59:10,447 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 485 ms to try 5000 different iterations.  Found a solution that moves 4706 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1767827760806043. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0404877359848966, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05235851546228312, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5228888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0404877359848966, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17216666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23683333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 02:59:10,473 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv538598167:81 , srv676587961:82 , srv1380490213:83 , srv1510573464:83 , srv1816080225:83 , srv2039624748:83 , srv278576726:83 , srv401210796:83 , srv611467337:83 , srv812115261:83 , srv822205035:83 , srv1360303677:84 , srv1361423653:84 , srv1509004861:84 , srv1569002140:84 , srv1873982448:84 , srv197346775:84 , srv289955177:84 , srv437227166:84 , srv455254178:84 , srv548168375:84 , srv626076108:84 , srv816538161:84 , srv1128205631:85 , srv1367802558:85 , srv1376891976:85 , srv1443575895:85 , srv1713889842:85 , srv1851652324:85 , srv1958123687:85 , srv1973996813:85 , srv204550218:85 , srv270211223:85 , srv290320267:85 , srv420592137:85 , srv477899241:85 , srv620034598:85 , srv1082290447:86 , srv115318733:86 , srv118070360:86 , srv1300857548:86 , srv1448856854:86 , srv1538970341:86 , srv1569727056:86 , srv1739144215:86 , srv1744560888:86 , srv1936090169:86 , srv556143086:86 , srv589766443:86 , srv600512498:86 , srv615772185:86 , srv746192214:86 , srv807853158:86 , srv889727896:86 , srv910038803:86 , srv1035545484:87 , srv1109635225:87 , srv1121509924:87 , srv1188998347:87 , srv1190716079:87 , srv14691775:87 , srv1552584077:87 , srv1778533678:87 , srv1892617476:87 , srv1894701879:87 , srv1900286260:87 , srv2133075822:87 , srv545308652:87 , srv639633096:87 , srv102449838:88 , srv1110330537:88 , srv1369039534:88 , srv1467253354:88 , srv1597494809:88 , srv1721830519:88 , srv1818525951:88 , srv26084775:88 , srv311284573:88 , srv414228734:88 , srv492242005:88 , srv110523231:89 , srv1439721216:89 , srv1911938:89 , srv204394430:89 , srv2068341176:89 , srv270373273:89 , srv284581471:89 , srv396147642:89 , srv485065905:89 , srv613684367:89 , srv1545984984:90 , srv1777497230:90 , srv1853690378:90 , srv1954317910:90 , srv1743558269:91 , srv1790273370:91 , srv245635075:91 , srv963008430:92 , srv2066664525:93 , srv698306428:450 }
2023-03-22 02:59:10,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-22 02:59:10,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1035545484=1, srv545308652=79, srv1110330537=5, srv1713889842=32, srv1873982448=44, srv284581471=65, srv115318733=8, srv600512498=83, srv1082290447=2, srv548168375=80, srv2133075822=59, srv1121509924=6, srv1892617476=45, srv1958123687=51, srv290320267=67, srv1744560888=36, srv204550218=56, srv620034598=87, srv1367802558=15, srv615772185=86, srv110523231=3, srv1448856854=21, srv613684367=85, srv1851652324=42, srv1894701879=46, srv437227166=73, srv485065905=76, srv401210796=70, srv812115261=94, srv1743558269=35, srv1376891976=17, srv1777497230=37, srv311284573=68, srv420592137=72, srv910038803=98, srv1190716079=11, srv289955177=66, srv1911938=48, srv807853158=93, srv1509004861=24, srv1569727056=30, srv589766443=82, srv1721830519=33, srv639633096=89, srv698306428=91, srv1109635225=4, srv626076108=88, srv538598167=78, srv1360303677=13, srv1552584077=28, srv204394430=55, srv26084775=61, srv1369039534=16, srv1954317910=50, srv676587961=90, srv245635075=60, srv1545984984=27, srv278576726=64, srv1439721216=19, srv1900286260=47, srv1818525951=41, srv1380490213=18, srv118070360=9, srv414228734=71, srv270211223=62, srv14691775=23, srv1778533678=38, srv102449838=0, srv270373273=63, srv455254178=74, srv1188998347=10, srv822205035=96, srv1597494809=31, srv1853690378=43, srv1361423653=14, srv2066664525=57, srv1300857548=12, srv1936090169=49, srv1816080225=40, srv1443575895=20, srv963008430=99, srv492242005=77, srv1467253354=22, srv889727896=97, srv1510573464=25, srv1790273370=39, srv396147642=69, srv1973996813=53, srv197346775=52, srv556143086=81, srv1569002140=29, srv1739144215=34, srv477899241=75, srv816538161=95, srv1538970341=26, srv611467337=84, srv1128205631=7, srv2068341176=58, srv2039624748=54, srv746192214=92} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-22 02:59:10,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-22 02:59:10,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-22 02:59:10,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-22 02:59:10,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-22 02:59:10,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-22 02:59:10,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-22 02:59:10,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-22 02:59:10,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-22 02:59:10,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-22 02:59:10,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-22 02:59:10,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-22 02:59:10,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-22 02:59:10,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-22 02:59:10,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-22 02:59:10,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-22 02:59:10,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-22 02:59:10,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-22 02:59:10,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-22 02:59:10,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-22 02:59:10,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-22 02:59:10,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-22 02:59:10,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-22 02:59:10,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-22 02:59:10,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-22 02:59:10,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-22 02:59:10,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-22 02:59:10,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-22 02:59:10,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-22 02:59:10,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-22 02:59:10,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-22 02:59:10,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-22 02:59:10,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-22 02:59:10,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-22 02:59:10,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-22 02:59:10,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-22 02:59:10,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-22 02:59:10,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-22 02:59:10,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-22 02:59:10,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-22 02:59:10,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-22 02:59:10,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-22 02:59:10,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-22 02:59:10,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-22 02:59:10,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-22 02:59:10,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-22 02:59:10,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-22 02:59:10,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-22 02:59:10,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-22 02:59:10,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-22 02:59:10,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-22 02:59:10,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-22 02:59:10,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-22 02:59:10,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-22 02:59:10,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-22 02:59:10,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-22 02:59:10,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-22 02:59:10,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-22 02:59:10,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-22 02:59:10,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-22 02:59:10,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-22 02:59:10,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-22 02:59:10,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-22 02:59:10,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-22 02:59:10,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-22 02:59:10,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-22 02:59:10,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-22 02:59:10,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-22 02:59:10,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-22 02:59:10,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-22 02:59:10,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-22 02:59:10,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-22 02:59:10,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-22 02:59:10,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-22 02:59:10,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-22 02:59:10,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-22 02:59:10,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-22 02:59:10,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-22 02:59:10,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-22 02:59:10,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-22 02:59:10,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-22 02:59:10,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-22 02:59:10,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-22 02:59:10,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-22 02:59:10,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-22 02:59:10,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-22 02:59:10,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-22 02:59:10,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-22 02:59:10,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-22 02:59:10,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-22 02:59:10,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-22 02:59:10,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-22 02:59:10,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-22 02:59:10,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-22 02:59:10,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-22 02:59:10,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-22 02:59:10,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-22 02:59:10,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-22 02:59:10,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-22 02:59:10,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-22 02:59:10,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-22 02:59:10,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-22 02:59:10,661 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0404877359848966, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05235851546228312, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0404877359848966, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17216666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23683333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 02:59:10,662 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-22 02:59:10,662 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1767498239643046, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0404877359848966, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05235851546228312, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0404877359848966, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17216666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23683333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-22 02:59:10,892 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 342 ms to try 5000 different iterations.  Found a solution that moves 2575 regions; Going from a computed imbalance of 0.1767498239643046 to a new imbalance of 0.013453438541832488. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022513245882803602); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.027307123838765573, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2861111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022513245882803602); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043333333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 02:59:10,903 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2066664525:77 , srv245635075:80 , srv963008430:80 , srv110523231:81 , srv1369039534:81 , srv1743558269:81 , srv1853690378:81 , srv1954317910:81 , srv284581471:81 , srv613684367:81 , srv1439721216:82 , srv1545984984:82 , srv1818525951:82 , srv102449838:83 , srv1188998347:83 , srv1777497230:83 , srv1790273370:83 , srv1900286260:83 , srv1911938:83 , srv26084775:83 , srv485065905:83 , srv807853158:83 , srv1035545484:84 , srv1110330537:84 , srv1121509924:84 , srv1448856854:84 , srv14691775:84 , srv1721830519:84 , srv1739144215:84 , srv2068341176:84 , srv270373273:84 , srv396147642:84 , srv589766443:84 , srv1082290447:85 , srv1128205631:85 , srv118070360:85 , srv1190716079:85 , srv1467253354:85 , srv1552584077:85 , srv1569727056:85 , srv204394430:85 , srv290320267:85 , srv311284573:85 , srv414228734:85 , srv746192214:85 , srv1109635225:86 , srv115318733:86 , srv1300857548:86 , srv1360303677:86 , srv1361423653:86 , srv1509004861:86 , srv1597494809:86 , srv1892617476:86 , srv1894701879:86 , srv1936090169:86 , srv197346775:86 , srv1973996813:86 , srv204550218:86 , srv270211223:86 , srv477899241:86 , srv545308652:86 , srv615772185:86 , srv910038803:86 , srv1443575895:87 , srv1538970341:87 , srv1713889842:87 , srv1744560888:87 , srv1778533678:87 , srv1851652324:87 , srv492242005:87 , srv556143086:87 , srv600512498:87 , srv639633096:87 , srv812115261:87 , srv1367802558:88 , srv1376891976:88 , srv1380490213:88 , srv1510573464:88 , srv1569002140:88 , srv1873982448:88 , srv620034598:88 , srv626076108:88 , srv822205035:88 , srv889727896:88 , srv2039624748:89 , srv2133075822:89 , srv278576726:89 , srv289955177:89 , srv401210796:89 , srv420592137:89 , srv548168375:89 , srv1816080225:90 , srv455254178:90 , srv538598167:90 , srv611467337:90 , srv816538161:90 , srv1958123687:91 , srv437227166:93 , srv676587961:93 , srv698306428:524 }
2023-03-22 02:59:10,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-22 02:59:10,911 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1035545484=1, srv545308652=79, srv1110330537=5, srv1713889842=32, srv1873982448=44, srv284581471=65, srv115318733=8, srv600512498=83, srv1082290447=2, srv548168375=80, srv2133075822=59, srv1121509924=6, srv1892617476=45, srv1958123687=51, srv290320267=67, srv1744560888=36, srv204550218=56, srv620034598=87, srv1367802558=15, srv615772185=86, srv110523231=3, srv1448856854=21, srv613684367=85, srv1851652324=42, srv1894701879=46, srv437227166=73, srv485065905=76, srv401210796=70, srv812115261=94, srv1743558269=35, srv1376891976=17, srv1777497230=37, srv311284573=68, srv420592137=72, srv910038803=98, srv1190716079=11, srv289955177=66, srv1911938=48, srv807853158=93, srv1509004861=24, srv1569727056=30, srv589766443=82, srv1721830519=33, srv639633096=89, srv698306428=91, srv1109635225=4, srv626076108=88, srv538598167=78, srv1360303677=13, srv1552584077=28, srv204394430=55, srv26084775=61, srv1369039534=16, srv1954317910=50, srv676587961=90, srv245635075=60, srv1545984984=27, srv278576726=64, srv1439721216=19, srv1900286260=47, srv1818525951=41, srv1380490213=18, srv118070360=9, srv414228734=71, srv270211223=62, srv14691775=23, srv1778533678=38, srv102449838=0, srv270373273=63, srv455254178=74, srv1188998347=10, srv822205035=96, srv1597494809=31, srv1853690378=43, srv1361423653=14, srv2066664525=57, srv1300857548=12, srv1936090169=49, srv1816080225=40, srv1443575895=20, srv963008430=99, srv492242005=77, srv1467253354=22, srv889727896=97, srv1510573464=25, srv1790273370=39, srv396147642=69, srv1973996813=53, srv197346775=52, srv556143086=81, srv1569002140=29, srv1739144215=34, srv477899241=75, srv816538161=95, srv1538970341=26, srv611467337=84, srv1128205631=7, srv2068341176=58, srv2039624748=54, srv746192214=92} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-22 02:59:10,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-22 02:59:10,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-22 02:59:10,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-22 02:59:10,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-22 02:59:10,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-22 02:59:10,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-22 02:59:10,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-22 02:59:10,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-22 02:59:10,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-22 02:59:10,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-22 02:59:10,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-22 02:59:10,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-22 02:59:10,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-22 02:59:10,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-22 02:59:10,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-22 02:59:10,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-22 02:59:10,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-22 02:59:10,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-22 02:59:10,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-22 02:59:10,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-22 02:59:10,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-22 02:59:10,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-22 02:59:10,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-22 02:59:10,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-22 02:59:10,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-22 02:59:10,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-22 02:59:10,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-22 02:59:10,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-22 02:59:10,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-22 02:59:10,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-22 02:59:10,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-22 02:59:10,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-22 02:59:10,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-22 02:59:10,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-22 02:59:10,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-22 02:59:10,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-22 02:59:10,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-22 02:59:10,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-22 02:59:10,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-22 02:59:10,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-22 02:59:10,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-22 02:59:10,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-22 02:59:10,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-22 02:59:10,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-22 02:59:10,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-22 02:59:10,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-22 02:59:10,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-22 02:59:10,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-22 02:59:10,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-22 02:59:10,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-22 02:59:10,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-22 02:59:10,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-22 02:59:10,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-22 02:59:10,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-22 02:59:10,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-22 02:59:10,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-22 02:59:10,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-22 02:59:10,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-22 02:59:10,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-22 02:59:10,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-22 02:59:10,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-22 02:59:10,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-22 02:59:10,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-22 02:59:10,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-22 02:59:10,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-22 02:59:10,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-22 02:59:10,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-22 02:59:10,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-22 02:59:10,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-22 02:59:10,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-22 02:59:10,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-22 02:59:10,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-22 02:59:10,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-22 02:59:10,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-22 02:59:10,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-22 02:59:10,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-22 02:59:10,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-22 02:59:10,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-22 02:59:10,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-22 02:59:10,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-22 02:59:10,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-22 02:59:10,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-22 02:59:10,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-22 02:59:10,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-22 02:59:10,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-22 02:59:10,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-22 02:59:10,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-22 02:59:10,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-22 02:59:10,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-22 02:59:10,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-22 02:59:10,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-22 02:59:10,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-22 02:59:10,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-22 02:59:10,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-22 02:59:10,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-22 02:59:10,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-22 02:59:10,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-22 02:59:10,945 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-22 02:59:10,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-22 02:59:10,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-22 02:59:10,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-22 02:59:11,066 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022513245882803602); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.027307123838765573, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022513245882803602); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043333333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 02:59:11,066 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-22 02:59:11,066 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013435408006458129, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022513245882803602); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.027307123838765573, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022513245882803602); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043333333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-22 02:59:11,234 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 264 ms to try 5000 different iterations.  Found a solution that moves 978 regions; Going from a computed imbalance of 0.013435408006458129 to a new imbalance of 5.925384310544117E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005980389451137794); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005381099233077657); MoveCostFunction : (multiplier=7.0, imbalance=0.10866666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005980389451137794); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 02:59:11,236 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv437227166:80 , srv1958123687:82 , srv2133075822:82 , srv676587961:82 , srv1597494809:83 , srv204394430:83 , srv420592137:83 , srv492242005:83 , srv1367802558:84 , srv1467253354:84 , srv1778533678:84 , srv1790273370:84 , srv1816080225:84 , srv1892617476:84 , srv1894701879:84 , srv289955177:84 , srv414228734:84 , srv545308652:84 , srv548168375:84 , srv639633096:84 , srv816538161:84 , srv889727896:84 , srv1109635225:85 , srv1190716079:85 , srv1300857548:85 , srv1376891976:85 , srv1538970341:85 , srv1545984984:85 , srv1552584077:85 , srv1569002140:85 , srv1713889842:85 , srv1743558269:85 , srv1744560888:85 , srv1777497230:85 , srv1851652324:85 , srv1873982448:85 , srv1911938:85 , srv2039624748:85 , srv2068341176:85 , srv270373273:85 , srv311284573:85 , srv396147642:85 , srv455254178:85 , srv485065905:85 , srv556143086:85 , srv600512498:85 , srv611467337:85 , srv615772185:85 , srv620034598:85 , srv626076108:85 , srv910038803:85 , srv102449838:86 , srv1035545484:86 , srv1082290447:86 , srv1110330537:86 , srv1121509924:86 , srv115318733:86 , srv1380490213:86 , srv14691775:86 , srv1510573464:86 , srv1569727056:86 , srv1721830519:86 , srv1853690378:86 , srv1936090169:86 , srv1954317910:86 , srv1973996813:86 , srv204550218:86 , srv270211223:86 , srv278576726:86 , srv401210796:86 , srv477899241:86 , srv746192214:86 , srv822205035:86 , srv963008430:86 , srv110523231:87 , srv1128205631:87 , srv118070360:87 , srv1361423653:87 , srv1439721216:87 , srv1443575895:87 , srv1448856854:87 , srv1509004861:87 , srv1739144215:87 , srv1900286260:87 , srv197346775:87 , srv2066664525:87 , srv245635075:87 , srv26084775:87 , srv538598167:87 , srv1188998347:88 , srv1360303677:88 , srv1369039534:88 , srv1818525951:88 , srv284581471:88 , srv290320267:88 , srv589766443:88 , srv613684367:88 , srv807853158:88 , srv812115261:89 , srv698306428:537 }
2023-03-22 02:59:11,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-22 02:59:11,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1035545484=1, srv545308652=79, srv1110330537=5, srv1713889842=32, srv1873982448=44, srv284581471=65, srv115318733=8, srv600512498=83, srv1082290447=2, srv548168375=80, srv2133075822=59, srv1121509924=6, srv1892617476=45, srv1958123687=51, srv290320267=67, srv1744560888=36, srv204550218=56, srv620034598=87, srv1367802558=15, srv615772185=86, srv110523231=3, srv1448856854=21, srv613684367=85, srv1851652324=42, srv1894701879=46, srv437227166=73, srv485065905=76, srv401210796=70, srv812115261=94, srv1743558269=35, srv1376891976=17, srv1777497230=37, srv311284573=68, srv420592137=72, srv910038803=98, srv1190716079=11, srv289955177=66, srv1911938=48, srv807853158=93, srv1509004861=24, srv1569727056=30, srv589766443=82, srv1721830519=33, srv639633096=89, srv698306428=91, srv1109635225=4, srv626076108=88, srv538598167=78, srv1360303677=13, srv1552584077=28, srv204394430=55, srv26084775=61, srv1369039534=16, srv1954317910=50, srv676587961=90, srv245635075=60, srv1545984984=27, srv278576726=64, srv1439721216=19, srv1900286260=47, srv1818525951=41, srv1380490213=18, srv118070360=9, srv414228734=71, srv270211223=62, srv14691775=23, srv1778533678=38, srv102449838=0, srv270373273=63, srv455254178=74, srv1188998347=10, srv822205035=96, srv1597494809=31, srv1853690378=43, srv1361423653=14, srv2066664525=57, srv1300857548=12, srv1936090169=49, srv1816080225=40, srv1443575895=20, srv963008430=99, srv492242005=77, srv1467253354=22, srv889727896=97, srv1510573464=25, srv1790273370=39, srv396147642=69, srv1973996813=53, srv197346775=52, srv556143086=81, srv1569002140=29, srv1739144215=34, srv477899241=75, srv816538161=95, srv1538970341=26, srv611467337=84, srv1128205631=7, srv2068341176=58, srv2039624748=54, srv746192214=92} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-22 02:59:11,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-22 02:59:11,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-22 02:59:11,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-22 02:59:11,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-22 02:59:11,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-22 02:59:11,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-22 02:59:11,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-22 02:59:11,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-22 02:59:11,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-22 02:59:11,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-22 02:59:11,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-22 02:59:11,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-22 02:59:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-22 02:59:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-22 02:59:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-22 02:59:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-22 02:59:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-22 02:59:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-22 02:59:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-22 02:59:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-22 02:59:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-22 02:59:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-22 02:59:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-22 02:59:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-22 02:59:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-22 02:59:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-22 02:59:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-22 02:59:11,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-22 02:59:11,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-22 02:59:11,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-22 02:59:11,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-22 02:59:11,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-22 02:59:11,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-22 02:59:11,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-22 02:59:11,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-22 02:59:11,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-22 02:59:11,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-22 02:59:11,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-22 02:59:11,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-22 02:59:11,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-22 02:59:11,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-22 02:59:11,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-22 02:59:11,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-22 02:59:11,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-22 02:59:11,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-22 02:59:11,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-22 02:59:11,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-22 02:59:11,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-22 02:59:11,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-22 02:59:11,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-22 02:59:11,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-22 02:59:11,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-22 02:59:11,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-22 02:59:11,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-22 02:59:11,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-22 02:59:11,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-22 02:59:11,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-22 02:59:11,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-22 02:59:11,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-22 02:59:11,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-22 02:59:11,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-22 02:59:11,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-22 02:59:11,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-22 02:59:11,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-22 02:59:11,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-22 02:59:11,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-22 02:59:11,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-22 02:59:11,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-22 02:59:11,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-22 02:59:11,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-22 02:59:11,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-22 02:59:11,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-22 02:59:11,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-22 02:59:11,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-22 02:59:11,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-22 02:59:11,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-22 02:59:11,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-22 02:59:11,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-22 02:59:11,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-22 02:59:11,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-22 02:59:11,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-22 02:59:11,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-22 02:59:11,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-22 02:59:11,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-22 02:59:11,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-22 02:59:11,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-22 02:59:11,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-22 02:59:11,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-22 02:59:11,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-22 02:59:11,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-22 02:59:11,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-22 02:59:11,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-22 02:59:11,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-22 02:59:11,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-22 02:59:11,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-22 02:59:11,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-22 02:59:11,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-22 02:59:11,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-22 02:59:11,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-22 02:59:11,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-22 02:59:11,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-22 02:59:11,375 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005980389451137794); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005381099233077657); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005980389451137794); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 02:59:11,375 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-22 02:59:11,375 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.856903286869849E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005980389451137794); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005381099233077657); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005980389451137794); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-22 02:59:11,621 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 349 ms to try 5000 different iterations.  Found a solution that moves 254 regions; Going from a computed imbalance of 5.856903286869849E-4 to a new imbalance of 2.0093445006846547E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.02822222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 02:59:11,622 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1958123687:83 , srv812115261:83 , srv963008430:83 , srv1109635225:84 , srv1110330537:84 , srv115318733:84 , srv118070360:84 , srv1190716079:84 , srv1360303677:84 , srv1367802558:84 , srv1376891976:84 , srv1443575895:84 , srv1538970341:84 , srv1569727056:84 , srv1597494809:84 , srv1721830519:84 , srv1744560888:84 , srv1777497230:84 , srv1778533678:84 , srv1790273370:84 , srv1818525951:84 , srv1936090169:84 , srv204394430:84 , srv204550218:84 , srv2068341176:84 , srv2133075822:84 , srv245635075:84 , srv26084775:84 , srv270373273:84 , srv278576726:84 , srv284581471:84 , srv311284573:84 , srv401210796:84 , srv455254178:84 , srv492242005:84 , srv538598167:84 , srv545308652:84 , srv556143086:84 , srv600512498:84 , srv611467337:84 , srv613684367:84 , srv620034598:84 , srv639633096:84 , srv816538161:84 , srv102449838:85 , srv1035545484:85 , srv1082290447:85 , srv110523231:85 , srv1121509924:85 , srv1128205631:85 , srv1188998347:85 , srv1300857548:85 , srv1361423653:85 , srv1369039534:85 , srv1380490213:85 , srv1439721216:85 , srv1448856854:85 , srv1467253354:85 , srv14691775:85 , srv1509004861:85 , srv1510573464:85 , srv1545984984:85 , srv1552584077:85 , srv1569002140:85 , srv1713889842:85 , srv1739144215:85 , srv1743558269:85 , srv1816080225:85 , srv1851652324:85 , srv1853690378:85 , srv1873982448:85 , srv1892617476:85 , srv1894701879:85 , srv1900286260:85 , srv1911938:85 , srv1954317910:85 , srv197346775:85 , srv1973996813:85 , srv2039624748:85 , srv2066664525:85 , srv270211223:85 , srv289955177:85 , srv290320267:85 , srv396147642:85 , srv414228734:85 , srv420592137:85 , srv437227166:85 , srv477899241:85 , srv548168375:85 , srv589766443:85 , srv615772185:85 , srv626076108:85 , srv676587961:85 , srv746192214:85 , srv807853158:85 , srv822205035:85 , srv889727896:85 , srv910038803:85 , srv485065905:86 , srv698306428:631 }
2023-03-22 02:59:11,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-22 02:59:11,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1035545484=1, srv545308652=79, srv1110330537=5, srv1713889842=32, srv1873982448=44, srv284581471=65, srv115318733=8, srv600512498=83, srv1082290447=2, srv548168375=80, srv2133075822=59, srv1121509924=6, srv1892617476=45, srv1958123687=51, srv290320267=67, srv1744560888=36, srv204550218=56, srv620034598=87, srv1367802558=15, srv615772185=86, srv110523231=3, srv1448856854=21, srv613684367=85, srv1851652324=42, srv1894701879=46, srv437227166=73, srv485065905=76, srv401210796=70, srv812115261=94, srv1743558269=35, srv1376891976=17, srv1777497230=37, srv311284573=68, srv420592137=72, srv910038803=98, srv1190716079=11, srv289955177=66, srv1911938=48, srv807853158=93, srv1509004861=24, srv1569727056=30, srv589766443=82, srv1721830519=33, srv639633096=89, srv698306428=91, srv1109635225=4, srv626076108=88, srv538598167=78, srv1360303677=13, srv1552584077=28, srv204394430=55, srv26084775=61, srv1369039534=16, srv1954317910=50, srv676587961=90, srv245635075=60, srv1545984984=27, srv278576726=64, srv1439721216=19, srv1900286260=47, srv1818525951=41, srv1380490213=18, srv118070360=9, srv414228734=71, srv270211223=62, srv14691775=23, srv1778533678=38, srv102449838=0, srv270373273=63, srv455254178=74, srv1188998347=10, srv822205035=96, srv1597494809=31, srv1853690378=43, srv1361423653=14, srv2066664525=57, srv1300857548=12, srv1936090169=49, srv1816080225=40, srv1443575895=20, srv963008430=99, srv492242005=77, srv1467253354=22, srv889727896=97, srv1510573464=25, srv1790273370=39, srv396147642=69, srv1973996813=53, srv197346775=52, srv556143086=81, srv1569002140=29, srv1739144215=34, srv477899241=75, srv816538161=95, srv1538970341=26, srv611467337=84, srv1128205631=7, srv2068341176=58, srv2039624748=54, srv746192214=92} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-22 02:59:11,642 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-22 02:59:11,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-22 02:59:11,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-22 02:59:11,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-22 02:59:11,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-22 02:59:11,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-22 02:59:11,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-22 02:59:11,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-22 02:59:11,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-22 02:59:11,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-22 02:59:11,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-22 02:59:11,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-22 02:59:11,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-22 02:59:11,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-22 02:59:11,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-22 02:59:11,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-22 02:59:11,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-22 02:59:11,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-22 02:59:11,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-22 02:59:11,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-22 02:59:11,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-22 02:59:11,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-22 02:59:11,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-22 02:59:11,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-22 02:59:11,644 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-22 02:59:11,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-22 02:59:11,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-22 02:59:11,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-22 02:59:11,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-22 02:59:11,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-22 02:59:11,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-22 02:59:11,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-22 02:59:11,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-22 02:59:11,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-22 02:59:11,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-22 02:59:11,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-22 02:59:11,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-22 02:59:11,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-22 02:59:11,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-22 02:59:11,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-22 02:59:11,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-22 02:59:11,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-22 02:59:11,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-22 02:59:11,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-22 02:59:11,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-22 02:59:11,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-22 02:59:11,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-22 02:59:11,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-22 02:59:11,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-22 02:59:11,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-22 02:59:11,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-22 02:59:11,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-22 02:59:11,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-22 02:59:11,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-22 02:59:11,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-22 02:59:11,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-22 02:59:11,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-22 02:59:11,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-22 02:59:11,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-22 02:59:11,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-22 02:59:11,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-22 02:59:11,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-22 02:59:11,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-22 02:59:11,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-22 02:59:11,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-22 02:59:11,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-22 02:59:11,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-22 02:59:11,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-22 02:59:11,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-22 02:59:11,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-22 02:59:11,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-22 02:59:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-22 02:59:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-22 02:59:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-22 02:59:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-22 02:59:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-22 02:59:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-22 02:59:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-22 02:59:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-22 02:59:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-22 02:59:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-22 02:59:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-22 02:59:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-22 02:59:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-22 02:59:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-22 02:59:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-22 02:59:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-22 02:59:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-22 02:59:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-22 02:59:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-22 02:59:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-22 02:59:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-22 02:59:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-22 02:59:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-22 02:59:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-22 02:59:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-22 02:59:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-22 02:59:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-22 02:59:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-22 02:59:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-22 02:59:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-22 02:59:11,728 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.8314898993220366E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 02:59:11,728 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1958123687:83 , srv812115261:83 , srv963008430:83 , srv1109635225:84 , srv1110330537:84 , srv115318733:84 , srv118070360:84 , srv1190716079:84 , srv1360303677:84 , srv1367802558:84 , srv1376891976:84 , srv1443575895:84 , srv1538970341:84 , srv1569727056:84 , srv1597494809:84 , srv1721830519:84 , srv1744560888:84 , srv1777497230:84 , srv1778533678:84 , srv1790273370:84 , srv1818525951:84 , srv1936090169:84 , srv204394430:84 , srv204550218:84 , srv2068341176:84 , srv2133075822:84 , srv245635075:84 , srv26084775:84 , srv270373273:84 , srv278576726:84 , srv284581471:84 , srv311284573:84 , srv401210796:84 , srv455254178:84 , srv492242005:84 , srv538598167:84 , srv545308652:84 , srv556143086:84 , srv600512498:84 , srv611467337:84 , srv613684367:84 , srv620034598:84 , srv639633096:84 , srv816538161:84 , srv102449838:85 , srv1035545484:85 , srv1082290447:85 , srv110523231:85 , srv1121509924:85 , srv1128205631:85 , srv1188998347:85 , srv1300857548:85 , srv1361423653:85 , srv1369039534:85 , srv1380490213:85 , srv1439721216:85 , srv1448856854:85 , srv1467253354:85 , srv14691775:85 , srv1509004861:85 , srv1510573464:85 , srv1545984984:85 , srv1552584077:85 , srv1569002140:85 , srv1713889842:85 , srv1739144215:85 , srv1743558269:85 , srv1816080225:85 , srv1851652324:85 , srv1853690378:85 , srv1873982448:85 , srv1892617476:85 , srv1894701879:85 , srv1900286260:85 , srv1911938:85 , srv1954317910:85 , srv197346775:85 , srv1973996813:85 , srv2039624748:85 , srv2066664525:85 , srv270211223:85 , srv289955177:85 , srv290320267:85 , srv396147642:85 , srv414228734:85 , srv420592137:85 , srv437227166:85 , srv477899241:85 , srv548168375:85 , srv589766443:85 , srv615772185:85 , srv626076108:85 , srv676587961:85 , srv746192214:85 , srv807853158:85 , srv822205035:85 , srv889727896:85 , srv910038803:85 , srv485065905:86 , srv698306428:631 }
2023-03-22 02:59:11,750 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=458 (was 458), ProcessCount=167 (was 167), AvailableMemoryMB=9455 (was 9968)
2023-03-22 02:59:11,756 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=458, ProcessCount=167, AvailableMemoryMB=9455
2023-03-22 02:59:11,757 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-22 02:59:11,757 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-03-22 02:59:11,768 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1003457067:84 , srv1009437772:84 , srv1127085889:84 , srv1169445293:84 , srv1175088256:84 , srv1187910560:84 , srv1468624096:84 , srv1537828693:84 , srv1637512080:84 , srv1683586034:84 , srv1729273222:84 , srv1760736433:84 , srv1890488593:84 , srv1894423435:84 , srv1949641481:84 , srv2016825327:84 , srv2119268552:84 , srv271952896:84 , srv36151256:84 , srv415000148:84 , srv536718754:84 , srv633264701:84 , srv688178102:84 , srv751475652:84 , srv828007787:84 , srv848887457:84 , srv869031350:84 , srv876136202:84 , srv936529296:84 , srv1805793610:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-03-22 02:59:11,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-22 02:59:11,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1760736433=11, srv1468624096=6, srv751475652=24, srv536718754=21, srv36151256=19, srv1537828693=7, srv1683586034=9, srv415000148=20, srv688178102=23, srv633264701=22, srv876136202=28, srv2016825327=16, srv1127085889=2, srv1729273222=10, srv1894423435=14, srv271952896=18, srv1187910560=5, srv1805793610=12, srv1003457067=0, srv2119268552=17, srv848887457=26, srv869031350=27, srv936529296=29, srv1169445293=3, srv1890488593=13, srv828007787=25, srv1175088256=4, srv1949641481=15, srv1009437772=1, srv1637512080=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-22 02:59:11,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-22 02:59:11,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-22 02:59:11,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-22 02:59:11,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-22 02:59:11,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-22 02:59:11,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-22 02:59:11,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-22 02:59:11,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-22 02:59:11,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-22 02:59:11,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-22 02:59:11,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-22 02:59:11,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-22 02:59:11,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-22 02:59:11,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-22 02:59:11,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-22 02:59:11,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-22 02:59:11,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-22 02:59:11,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-22 02:59:11,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-22 02:59:11,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-22 02:59:11,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-22 02:59:11,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-22 02:59:11,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-22 02:59:11,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-22 02:59:11,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-22 02:59:11,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-22 02:59:11,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-22 02:59:11,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-22 02:59:11,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-22 02:59:11,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-22 02:59:11,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-22 02:59:11,805 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1030426801494122, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 02:59:11,805 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912517727384779, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1030426801494122, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-03-22 03:01:11,791 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9639753 different iterations.  Found a solution that moves 1843 regions; Going from a computed imbalance of 0.9912517727384779 to a new imbalance of 4.3016539410932495E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6825925925925926, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 03:01:11,796 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1003457067:90 , srv1009437772:90 , srv1127085889:90 , srv1169445293:90 , srv1175088256:90 , srv1187910560:90 , srv1468624096:90 , srv1537828693:90 , srv1637512080:90 , srv1683586034:90 , srv1729273222:90 , srv1760736433:90 , srv1805793610:90 , srv1890488593:90 , srv1894423435:90 , srv1949641481:90 , srv2016825327:90 , srv2119268552:90 , srv271952896:90 , srv36151256:90 , srv415000148:90 , srv536718754:90 , srv633264701:90 , srv688178102:90 , srv751475652:90 , srv828007787:90 , srv848887457:90 , srv869031350:90 , srv876136202:90 , srv936529296:90 }
2023-03-22 03:01:11,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-22 03:01:11,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1760736433=11, srv1468624096=6, srv751475652=24, srv536718754=21, srv36151256=19, srv1537828693=7, srv1683586034=9, srv415000148=20, srv688178102=23, srv633264701=22, srv876136202=28, srv2016825327=16, srv1127085889=2, srv1729273222=10, srv1894423435=14, srv271952896=18, srv1187910560=5, srv1805793610=12, srv1003457067=0, srv2119268552=17, srv848887457=26, srv869031350=27, srv936529296=29, srv1169445293=3, srv1890488593=13, srv828007787=25, srv1175088256=4, srv1949641481=15, srv1009437772=1, srv1637512080=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-22 03:01:11,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-22 03:01:11,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-22 03:01:11,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-22 03:01:11,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-22 03:01:11,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-22 03:01:11,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-22 03:01:11,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-22 03:01:11,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-22 03:01:11,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-22 03:01:11,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-22 03:01:11,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-22 03:01:11,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-22 03:01:11,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-22 03:01:11,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-22 03:01:11,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-22 03:01:11,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-22 03:01:11,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-22 03:01:11,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-22 03:01:11,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-22 03:01:11,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-22 03:01:11,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-22 03:01:11,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-22 03:01:11,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-22 03:01:11,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-22 03:01:11,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-22 03:01:11,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-22 03:01:11,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-22 03:01:11,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-22 03:01:11,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-22 03:01:11,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-22 03:01:11,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-22 03:01:11,810 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-22 03:01:11,810 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1003457067:90 , srv1009437772:90 , srv1127085889:90 , srv1169445293:90 , srv1175088256:90 , srv1187910560:90 , srv1468624096:90 , srv1537828693:90 , srv1637512080:90 , srv1683586034:90 , srv1729273222:90 , srv1760736433:90 , srv1805793610:90 , srv1890488593:90 , srv1894423435:90 , srv1949641481:90 , srv2016825327:90 , srv2119268552:90 , srv271952896:90 , srv36151256:90 , srv415000148:90 , srv536718754:90 , srv633264701:90 , srv688178102:90 , srv751475652:90 , srv828007787:90 , srv848887457:90 , srv869031350:90 , srv876136202:90 , srv936529296:90 }
2023-03-22 03:01:11,821 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=200 (was 458), ProcessCount=163 (was 167), AvailableMemoryMB=10199 (was 9455) - AvailableMemoryMB LEAK? -