2023-04-04 17:57:12,550 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-04-04 17:57:12,810 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-04-04 17:57:12,907 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-04-04 17:57:12,958 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-04 17:57:12,965 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-04 17:57:12,978 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=282, ProcessCount=171, AvailableMemoryMB=8124
2023-04-04 17:57:12,980 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-04 17:57:12,980 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-04 17:57:13,083 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1016373651:84 , srv1024080925:84 , srv1053848651:84 , srv1058874041:84 , srv1068812114:84 , srv1083809150:84 , srv1125439563:84 , srv1135866950:84 , srv1145582238:84 , srv1165410730:84 , srv116656732:84 , srv1181992614:84 , srv1187450016:84 , srv1201225717:84 , srv1203442755:84 , srv1215770786:84 , srv1219518025:84 , srv1222324183:84 , srv1252553261:84 , srv1289721902:84 , srv133388558:84 , srv1371416379:84 , srv1394588950:84 , srv1403859699:84 , srv1408277952:84 , srv1421319842:84 , srv1430857813:84 , srv1455744920:84 , srv1488602806:84 , srv1510213411:84 , srv1510910502:84 , srv1532623531:84 , srv1532628946:84 , srv155741663:84 , srv1557461030:84 , srv161603882:84 , srv1652991268:84 , srv1676566076:84 , srv1682106945:84 , srv1716698314:84 , srv1727403267:84 , srv1731925965:84 , srv1732778173:84 , srv1767900481:84 , srv1780882452:84 , srv1801236030:84 , srv1818501412:84 , srv1835986763:84 , srv1900202198:84 , srv1911377082:84 , srv1933428091:84 , srv1945872937:84 , srv1951791612:84 , srv2027149052:84 , srv2040198677:84 , srv2057786855:84 , srv2064678088:84 , srv2096912194:84 , srv2117757312:84 , srv2139476480:84 , srv214192825:84 , srv228127437:84 , srv242164726:84 , srv299778329:84 , srv305122922:84 , srv336585116:84 , srv339484475:84 , srv345900100:84 , srv362074472:84 , srv372077928:84 , srv401799100:84 , srv404059649:84 , srv439430690:84 , srv445367394:84 , srv490875092:84 , srv504634409:84 , srv529364755:84 , srv537372672:84 , srv540016811:84 , srv652485976:84 , srv679673696:84 , srv730556340:84 , srv733020029:84 , srv763791921:84 , srv782113315:84 , srv787019510:84 , srv792818731:84 , srv802898905:84 , srv844555959:84 , srv886640142:84 , srv896840150:84 , srv92354283:84 , srv959018295:84 , srv96355184:84 , srv970492173:84 , srv975323362:84 , srv987585921:84 , srv99254213:84 , srv996566379:84 , srv1604525141:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-04-04 17:57:13,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-04 17:57:13,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1732778173=43, srv1371416379=21, srv2027149052=54, srv490875092=75, srv1652991268=37, srv996566379=99, srv1083809150=5, srv1222324183=17, srv336585116=66, srv1532628946=32, srv155741663=33, srv1731925965=42, srv362074472=69, srv1289721902=19, srv1532623531=31, srv1801236030=46, srv1900202198=49, srv2040198677=55, srv792818731=87, srv537372672=78, srv1394588950=22, srv1215770786=15, srv228127437=62, srv1024080925=1, srv133388558=20, srv1488602806=28, srv787019510=86, srv540016811=79, srv1219518025=16, srv1945872937=52, srv1016373651=0, srv1181992614=11, srv99254213=98, srv1165410730=9, srv1716698314=40, srv679673696=81, srv404059649=72, srv2139476480=60, srv504634409=76, srv445367394=74, srv1676566076=38, srv1201225717=13, srv1135866950=7, srv372077928=70, srv161603882=36, srv733020029=83, srv529364755=77, srv652485976=80, srv1145582238=8, srv1911377082=50, srv2057786855=56, srv1408277952=24, srv896840150=91, srv1835986763=48, srv242164726=63, srv92354283=92, srv1682106945=39, srv959018295=93, srv1951791612=53, srv1403859699=23, srv802898905=88, srv401799100=71, srv1421319842=25, srv299778329=64, srv970492173=95, srv1604525141=35, srv1727403267=41, srv1933428091=51, srv305122922=65, srv345900100=68, srv1455744920=27, srv2117757312=59, srv439430690=73, srv214192825=61, srv1187450016=12, srv763791921=84, srv1430857813=26, srv1510213411=29, srv1557461030=34, srv1068812114=4, srv1053848651=2, srv339484475=67, srv1203442755=14, srv2096912194=58, srv975323362=96, srv1058874041=3, srv844555959=89, srv1125439563=6, srv782113315=85, srv1767900481=44, srv1818501412=47, srv116656732=10, srv1510910502=30, srv96355184=94, srv987585921=97, srv2064678088=57, srv1252553261=18, srv730556340=82, srv1780882452=45, srv886640142=90} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-04 17:57:13,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-04 17:57:13,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-04 17:57:13,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-04 17:57:13,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-04 17:57:13,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-04 17:57:13,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-04 17:57:13,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-04 17:57:13,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-04 17:57:13,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-04 17:57:13,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-04 17:57:13,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-04 17:57:13,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-04 17:57:13,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-04 17:57:13,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-04 17:57:13,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-04 17:57:13,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-04 17:57:13,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-04 17:57:13,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-04 17:57:13,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-04 17:57:13,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-04 17:57:13,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-04 17:57:13,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-04 17:57:13,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-04 17:57:13,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-04 17:57:13,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-04 17:57:13,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-04 17:57:13,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-04 17:57:13,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-04 17:57:13,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-04 17:57:13,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-04 17:57:13,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-04 17:57:13,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-04 17:57:13,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-04 17:57:13,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-04 17:57:13,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-04 17:57:13,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-04 17:57:13,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-04 17:57:13,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-04 17:57:13,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-04 17:57:13,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-04 17:57:13,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-04 17:57:13,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-04 17:57:13,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-04 17:57:13,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-04 17:57:13,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-04 17:57:13,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-04 17:57:13,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-04 17:57:13,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-04 17:57:13,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-04 17:57:13,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-04 17:57:13,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-04 17:57:13,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-04 17:57:13,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-04 17:57:13,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-04 17:57:13,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-04 17:57:13,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-04 17:57:13,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-04 17:57:13,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-04 17:57:13,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-04 17:57:13,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-04 17:57:13,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-04 17:57:13,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-04 17:57:13,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-04 17:57:13,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-04 17:57:13,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-04 17:57:13,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-04 17:57:13,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-04 17:57:13,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-04 17:57:13,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-04 17:57:13,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-04 17:57:13,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-04 17:57:13,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-04 17:57:13,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-04 17:57:13,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-04 17:57:13,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-04 17:57:13,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-04 17:57:13,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-04 17:57:13,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-04 17:57:13,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-04 17:57:13,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-04 17:57:13,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-04 17:57:13,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-04 17:57:13,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-04 17:57:13,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-04 17:57:13,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-04 17:57:13,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-04 17:57:13,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-04 17:57:13,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-04 17:57:13,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-04 17:57:13,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-04 17:57:13,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-04 17:57:13,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-04 17:57:13,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-04 17:57:13,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-04 17:57:13,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-04 17:57:13,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-04 17:57:13,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-04 17:57:13,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-04 17:57:13,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-04 17:57:13,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-04 17:57:13,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-04 17:57:13,357 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-04 17:57:13,357 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-04 17:57:13,357 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-04 17:57:13,639 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 429 ms to try 5000 different iterations.  Found a solution that moves 4688 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17380403028355318. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040348278018952057, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055795441687751815, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5208888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040348278018952057, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16925, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-04 17:57:13,653 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv214192825:81 , srv679673696:82 , srv1053848651:83 , srv1165410730:83 , srv1203442755:83 , srv1222324183:83 , srv1510213411:83 , srv1900202198:83 , srv1911377082:83 , srv1951791612:83 , srv2057786855:83 , srv345900100:83 , srv404059649:83 , srv763791921:83 , srv844555959:83 , srv1068812114:84 , srv1125439563:84 , srv1135866950:84 , srv1145582238:84 , srv1201225717:84 , srv1394588950:84 , srv161603882:84 , srv1767900481:84 , srv1801236030:84 , srv1835986763:84 , srv1945872937:84 , srv2139476480:84 , srv305122922:84 , srv959018295:84 , srv987585921:84 , srv996566379:84 , srv1252553261:85 , srv1421319842:85 , srv1557461030:85 , srv1732778173:85 , srv1780882452:85 , srv1818501412:85 , srv2027149052:85 , srv228127437:85 , srv339484475:85 , srv445367394:85 , srv886640142:85 , srv92354283:85 , srv96355184:85 , srv1058874041:86 , srv1083809150:86 , srv1371416379:86 , srv1403859699:86 , srv1455744920:86 , srv1532628946:86 , srv155741663:86 , srv1676566076:86 , srv1731925965:86 , srv2040198677:86 , srv2064678088:86 , srv299778329:86 , srv336585116:86 , srv372077928:86 , srv652485976:86 , srv733020029:86 , srv792818731:86 , srv802898905:86 , srv975323362:86 , srv1016373651:87 , srv1181992614:87 , srv1430857813:87 , srv1532623531:87 , srv1652991268:87 , srv362074472:87 , srv504634409:87 , srv730556340:87 , srv896840150:87 , srv1024080925:88 , srv1408277952:88 , srv1488602806:88 , srv1510910502:88 , srv1682106945:88 , srv1727403267:88 , srv1933428091:88 , srv2117757312:88 , srv242164726:88 , srv439430690:88 , srv529364755:88 , srv537372672:88 , srv540016811:88 , srv787019510:88 , srv401799100:89 , srv1187450016:90 , srv1219518025:90 , srv782113315:90 , srv1289721902:91 , srv2096912194:91 , srv133388558:93 , srv99254213:94 , srv970492173:96 , srv116656732:97 , srv1215770786:97 , srv1716698314:97 , srv490875092:97 , srv1604525141:448 }
2023-04-04 17:57:13,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-04 17:57:13,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1732778173=43, srv1371416379=21, srv2027149052=54, srv490875092=75, srv1652991268=37, srv996566379=99, srv1083809150=5, srv1222324183=17, srv336585116=66, srv1532628946=32, srv155741663=33, srv1731925965=42, srv362074472=69, srv1289721902=19, srv1532623531=31, srv1801236030=46, srv1900202198=49, srv2040198677=55, srv792818731=87, srv537372672=78, srv1394588950=22, srv1215770786=15, srv228127437=62, srv1024080925=1, srv133388558=20, srv1488602806=28, srv787019510=86, srv540016811=79, srv1219518025=16, srv1945872937=52, srv1016373651=0, srv1181992614=11, srv99254213=98, srv1165410730=9, srv1716698314=40, srv679673696=81, srv404059649=72, srv2139476480=60, srv504634409=76, srv445367394=74, srv1676566076=38, srv1201225717=13, srv1135866950=7, srv372077928=70, srv161603882=36, srv733020029=83, srv529364755=77, srv652485976=80, srv1145582238=8, srv1911377082=50, srv2057786855=56, srv1408277952=24, srv896840150=91, srv1835986763=48, srv242164726=63, srv92354283=92, srv1682106945=39, srv959018295=93, srv1951791612=53, srv1403859699=23, srv802898905=88, srv401799100=71, srv1421319842=25, srv299778329=64, srv970492173=95, srv1604525141=35, srv1727403267=41, srv1933428091=51, srv305122922=65, srv345900100=68, srv1455744920=27, srv2117757312=59, srv439430690=73, srv214192825=61, srv1187450016=12, srv763791921=84, srv1430857813=26, srv1510213411=29, srv1557461030=34, srv1068812114=4, srv1053848651=2, srv339484475=67, srv1203442755=14, srv2096912194=58, srv975323362=96, srv1058874041=3, srv844555959=89, srv1125439563=6, srv782113315=85, srv1767900481=44, srv1818501412=47, srv116656732=10, srv1510910502=30, srv96355184=94, srv987585921=97, srv2064678088=57, srv1252553261=18, srv730556340=82, srv1780882452=45, srv886640142=90} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-04 17:57:13,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-04 17:57:13,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-04 17:57:13,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-04 17:57:13,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-04 17:57:13,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-04 17:57:13,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-04 17:57:13,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-04 17:57:13,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-04 17:57:13,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-04 17:57:13,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-04 17:57:13,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-04 17:57:13,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-04 17:57:13,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-04 17:57:13,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-04 17:57:13,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-04 17:57:13,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-04 17:57:13,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-04 17:57:13,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-04 17:57:13,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-04 17:57:13,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-04 17:57:13,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-04 17:57:13,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-04 17:57:13,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-04 17:57:13,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-04 17:57:13,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-04 17:57:13,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-04 17:57:13,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-04 17:57:13,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-04 17:57:13,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-04 17:57:13,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-04 17:57:13,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-04 17:57:13,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-04 17:57:13,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-04 17:57:13,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-04 17:57:13,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-04 17:57:13,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-04 17:57:13,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-04 17:57:13,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-04 17:57:13,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-04 17:57:13,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-04 17:57:13,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-04 17:57:13,688 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-04 17:57:13,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-04 17:57:13,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-04 17:57:13,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-04 17:57:13,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-04 17:57:13,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-04 17:57:13,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-04 17:57:13,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-04 17:57:13,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-04 17:57:13,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-04 17:57:13,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-04 17:57:13,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-04 17:57:13,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-04 17:57:13,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-04 17:57:13,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-04 17:57:13,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-04 17:57:13,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-04 17:57:13,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-04 17:57:13,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-04 17:57:13,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-04 17:57:13,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-04 17:57:13,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-04 17:57:13,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-04 17:57:13,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-04 17:57:13,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-04 17:57:13,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-04 17:57:13,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-04 17:57:13,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-04 17:57:13,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-04 17:57:13,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-04 17:57:13,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-04 17:57:13,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-04 17:57:13,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-04 17:57:13,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-04 17:57:13,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-04 17:57:13,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-04 17:57:13,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-04 17:57:13,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-04 17:57:13,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-04 17:57:13,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-04 17:57:13,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-04 17:57:13,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-04 17:57:13,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-04 17:57:13,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-04 17:57:13,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-04 17:57:13,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-04 17:57:13,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-04 17:57:13,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-04 17:57:13,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-04 17:57:13,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-04 17:57:13,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-04 17:57:13,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-04 17:57:13,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-04 17:57:13,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-04 17:57:13,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-04 17:57:13,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-04 17:57:13,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-04 17:57:13,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-04 17:57:13,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-04 17:57:13,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-04 17:57:13,847 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040348278018952057, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055795441687751815, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040348278018952057, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16925, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-04 17:57:13,847 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-04 17:57:13,847 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17377120420594735, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040348278018952057, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055795441687751815, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040348278018952057, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16925, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-04 17:57:14,034 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 313 ms to try 5000 different iterations.  Found a solution that moves 2613 regions; Going from a computed imbalance of 0.17377120420594735 to a new imbalance of 0.016369402787215933. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021442551511855364); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031186776673483043, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29033333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021442551511855364); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04475, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-04 17:57:14,043 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1716698314:75 , srv1215770786:76 , srv490875092:77 , srv116656732:79 , srv99254213:79 , srv970492173:80 , srv133388558:81 , srv2096912194:81 , srv1187450016:82 , srv1289721902:82 , srv439430690:82 , srv730556340:82 , srv782113315:82 , srv896840150:82 , srv1024080925:83 , srv1394588950:83 , srv1488602806:83 , srv1727403267:83 , srv242164726:83 , srv401799100:83 , srv529364755:83 , srv1016373651:84 , srv1219518025:84 , srv1408277952:84 , srv1532623531:84 , srv1682106945:84 , srv1933428091:84 , srv2040198677:84 , srv2117757312:84 , srv504634409:84 , srv537372672:84 , srv787019510:84 , srv802898905:84 , srv1201225717:85 , srv1421319842:85 , srv1455744920:85 , srv1780882452:85 , srv228127437:85 , srv305122922:85 , srv336585116:85 , srv733020029:85 , srv792818731:85 , srv987585921:85 , srv1058874041:86 , srv1125439563:86 , srv1135866950:86 , srv1181992614:86 , srv1252553261:86 , srv1371416379:86 , srv1403859699:86 , srv1510910502:86 , srv161603882:86 , srv1767900481:86 , srv1835986763:86 , srv1951791612:86 , srv372077928:86 , srv540016811:86 , srv975323362:86 , srv1083809150:87 , srv1203442755:87 , srv1430857813:87 , srv1532628946:87 , srv1652991268:87 , srv1731925965:87 , srv1801236030:87 , srv1945872937:87 , srv2064678088:87 , srv339484475:87 , srv362074472:87 , srv404059649:87 , srv652485976:87 , srv763791921:87 , srv886640142:87 , srv92354283:87 , srv96355184:87 , srv1068812114:88 , srv1145582238:88 , srv1165410730:88 , srv1222324183:88 , srv1732778173:88 , srv1818501412:88 , srv2027149052:88 , srv2139476480:88 , srv445367394:88 , srv1053848651:89 , srv155741663:89 , srv1557461030:89 , srv1676566076:89 , srv214192825:89 , srv1911377082:90 , srv299778329:90 , srv679673696:90 , srv1510213411:91 , srv1900202198:91 , srv959018295:91 , srv2057786855:93 , srv345900100:93 , srv844555959:95 , srv996566379:95 , srv1604525141:516 }
2023-04-04 17:57:14,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-04 17:57:14,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1732778173=43, srv1371416379=21, srv2027149052=54, srv490875092=75, srv1652991268=37, srv996566379=99, srv1083809150=5, srv1222324183=17, srv336585116=66, srv1532628946=32, srv155741663=33, srv1731925965=42, srv362074472=69, srv1289721902=19, srv1532623531=31, srv1801236030=46, srv1900202198=49, srv2040198677=55, srv792818731=87, srv537372672=78, srv1394588950=22, srv1215770786=15, srv228127437=62, srv1024080925=1, srv133388558=20, srv1488602806=28, srv787019510=86, srv540016811=79, srv1219518025=16, srv1945872937=52, srv1016373651=0, srv1181992614=11, srv99254213=98, srv1165410730=9, srv1716698314=40, srv679673696=81, srv404059649=72, srv2139476480=60, srv504634409=76, srv445367394=74, srv1676566076=38, srv1201225717=13, srv1135866950=7, srv372077928=70, srv161603882=36, srv733020029=83, srv529364755=77, srv652485976=80, srv1145582238=8, srv1911377082=50, srv2057786855=56, srv1408277952=24, srv896840150=91, srv1835986763=48, srv242164726=63, srv92354283=92, srv1682106945=39, srv959018295=93, srv1951791612=53, srv1403859699=23, srv802898905=88, srv401799100=71, srv1421319842=25, srv299778329=64, srv970492173=95, srv1604525141=35, srv1727403267=41, srv1933428091=51, srv305122922=65, srv345900100=68, srv1455744920=27, srv2117757312=59, srv439430690=73, srv214192825=61, srv1187450016=12, srv763791921=84, srv1430857813=26, srv1510213411=29, srv1557461030=34, srv1068812114=4, srv1053848651=2, srv339484475=67, srv1203442755=14, srv2096912194=58, srv975323362=96, srv1058874041=3, srv844555959=89, srv1125439563=6, srv782113315=85, srv1767900481=44, srv1818501412=47, srv116656732=10, srv1510910502=30, srv96355184=94, srv987585921=97, srv2064678088=57, srv1252553261=18, srv730556340=82, srv1780882452=45, srv886640142=90} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-04 17:57:14,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-04 17:57:14,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-04 17:57:14,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-04 17:57:14,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-04 17:57:14,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-04 17:57:14,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-04 17:57:14,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-04 17:57:14,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-04 17:57:14,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-04 17:57:14,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-04 17:57:14,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-04 17:57:14,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-04 17:57:14,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-04 17:57:14,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-04 17:57:14,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-04 17:57:14,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-04 17:57:14,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-04 17:57:14,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-04 17:57:14,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-04 17:57:14,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-04 17:57:14,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-04 17:57:14,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-04 17:57:14,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-04 17:57:14,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-04 17:57:14,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-04 17:57:14,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-04 17:57:14,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-04 17:57:14,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-04 17:57:14,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-04 17:57:14,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-04 17:57:14,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-04 17:57:14,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-04 17:57:14,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-04 17:57:14,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-04 17:57:14,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-04 17:57:14,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-04 17:57:14,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-04 17:57:14,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-04 17:57:14,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-04 17:57:14,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-04 17:57:14,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-04 17:57:14,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-04 17:57:14,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-04 17:57:14,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-04 17:57:14,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-04 17:57:14,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-04 17:57:14,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-04 17:57:14,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-04 17:57:14,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-04 17:57:14,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-04 17:57:14,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-04 17:57:14,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-04 17:57:14,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-04 17:57:14,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-04 17:57:14,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-04 17:57:14,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-04 17:57:14,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-04 17:57:14,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-04 17:57:14,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-04 17:57:14,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-04 17:57:14,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-04 17:57:14,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-04 17:57:14,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-04 17:57:14,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-04 17:57:14,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-04 17:57:14,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-04 17:57:14,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-04 17:57:14,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-04 17:57:14,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-04 17:57:14,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-04 17:57:14,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-04 17:57:14,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-04 17:57:14,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-04 17:57:14,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-04 17:57:14,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-04 17:57:14,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-04 17:57:14,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-04 17:57:14,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-04 17:57:14,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-04 17:57:14,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-04 17:57:14,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-04 17:57:14,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-04 17:57:14,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-04 17:57:14,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-04 17:57:14,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-04 17:57:14,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-04 17:57:14,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-04 17:57:14,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-04 17:57:14,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-04 17:57:14,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-04 17:57:14,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-04 17:57:14,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-04 17:57:14,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-04 17:57:14,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-04 17:57:14,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-04 17:57:14,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-04 17:57:14,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-04 17:57:14,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-04 17:57:14,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-04 17:57:14,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-04 17:57:14,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-04 17:57:14,209 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021442551511855364); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031186776673483043, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021442551511855364); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04475, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-04 17:57:14,209 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-04 17:57:14,209 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016351106170154495, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021442551511855364); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031186776673483043, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021442551511855364); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04475, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-04 17:57:14,377 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 291 ms to try 5000 different iterations.  Found a solution that moves 1009 regions; Going from a computed imbalance of 0.016351106170154495 to a new imbalance of 9.983308246301508E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006321005190772531); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0067834882385377025); MoveCostFunction : (multiplier=7.0, imbalance=0.1121111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006321005190772531); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0045); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-04 17:57:14,378 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv996566379:78 , srv844555959:80 , srv2057786855:81 , srv299778329:81 , srv970492173:81 , srv116656732:82 , srv1676566076:82 , srv345900100:82 , srv1430857813:83 , srv1510213411:83 , srv155741663:83 , srv1900202198:83 , srv362074472:83 , srv490875092:83 , srv540016811:83 , srv959018295:83 , srv1181992614:84 , srv1215770786:84 , srv1219518025:84 , srv1289721902:84 , srv133388558:84 , srv1510910502:84 , srv1532628946:84 , srv1652991268:84 , srv1732778173:84 , srv1818501412:84 , srv1911377082:84 , srv2027149052:84 , srv1053848651:85 , srv1058874041:85 , srv1068812114:85 , srv1145582238:85 , srv1187450016:85 , srv1371416379:85 , srv1455744920:85 , srv1557461030:85 , srv1682106945:85 , srv1731925965:85 , srv2117757312:85 , srv2139476480:85 , srv339484475:85 , srv372077928:85 , srv401799100:85 , srv445367394:85 , srv787019510:85 , srv886640142:85 , srv96355184:85 , srv975323362:85 , srv99254213:85 , srv1016373651:86 , srv1083809150:86 , srv1165410730:86 , srv1222324183:86 , srv1252553261:86 , srv1403859699:86 , srv1408277952:86 , srv1488602806:86 , srv1716698314:86 , srv1801236030:86 , srv1933428091:86 , srv1945872937:86 , srv2064678088:86 , srv2096912194:86 , srv242164726:86 , srv404059649:86 , srv504634409:86 , srv537372672:86 , srv652485976:86 , srv679673696:86 , srv733020029:86 , srv782113315:86 , srv792818731:86 , srv92354283:86 , srv1024080925:87 , srv1125439563:87 , srv1203442755:87 , srv1421319842:87 , srv1532623531:87 , srv1727403267:87 , srv1835986763:87 , srv2040198677:87 , srv214192825:87 , srv336585116:87 , srv529364755:87 , srv763791921:87 , srv1135866950:88 , srv161603882:88 , srv1767900481:88 , srv1780882452:88 , srv228127437:88 , srv439430690:88 , srv730556340:88 , srv802898905:88 , srv896840150:88 , srv1201225717:89 , srv1951791612:89 , srv305122922:89 , srv987585921:89 , srv1394588950:90 , srv1604525141:550 }
2023-04-04 17:57:14,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-04 17:57:14,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1732778173=43, srv1371416379=21, srv2027149052=54, srv490875092=75, srv1652991268=37, srv996566379=99, srv1083809150=5, srv1222324183=17, srv336585116=66, srv1532628946=32, srv155741663=33, srv1731925965=42, srv362074472=69, srv1289721902=19, srv1532623531=31, srv1801236030=46, srv1900202198=49, srv2040198677=55, srv792818731=87, srv537372672=78, srv1394588950=22, srv1215770786=15, srv228127437=62, srv1024080925=1, srv133388558=20, srv1488602806=28, srv787019510=86, srv540016811=79, srv1219518025=16, srv1945872937=52, srv1016373651=0, srv1181992614=11, srv99254213=98, srv1165410730=9, srv1716698314=40, srv679673696=81, srv404059649=72, srv2139476480=60, srv504634409=76, srv445367394=74, srv1676566076=38, srv1201225717=13, srv1135866950=7, srv372077928=70, srv161603882=36, srv733020029=83, srv529364755=77, srv652485976=80, srv1145582238=8, srv1911377082=50, srv2057786855=56, srv1408277952=24, srv896840150=91, srv1835986763=48, srv242164726=63, srv92354283=92, srv1682106945=39, srv959018295=93, srv1951791612=53, srv1403859699=23, srv802898905=88, srv401799100=71, srv1421319842=25, srv299778329=64, srv970492173=95, srv1604525141=35, srv1727403267=41, srv1933428091=51, srv305122922=65, srv345900100=68, srv1455744920=27, srv2117757312=59, srv439430690=73, srv214192825=61, srv1187450016=12, srv763791921=84, srv1430857813=26, srv1510213411=29, srv1557461030=34, srv1068812114=4, srv1053848651=2, srv339484475=67, srv1203442755=14, srv2096912194=58, srv975323362=96, srv1058874041=3, srv844555959=89, srv1125439563=6, srv782113315=85, srv1767900481=44, srv1818501412=47, srv116656732=10, srv1510910502=30, srv96355184=94, srv987585921=97, srv2064678088=57, srv1252553261=18, srv730556340=82, srv1780882452=45, srv886640142=90} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-04 17:57:14,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-04 17:57:14,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-04 17:57:14,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-04 17:57:14,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-04 17:57:14,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-04 17:57:14,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-04 17:57:14,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-04 17:57:14,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-04 17:57:14,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-04 17:57:14,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-04 17:57:14,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-04 17:57:14,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-04 17:57:14,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-04 17:57:14,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-04 17:57:14,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-04 17:57:14,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-04 17:57:14,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-04 17:57:14,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-04 17:57:14,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-04 17:57:14,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-04 17:57:14,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-04 17:57:14,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-04 17:57:14,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-04 17:57:14,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-04 17:57:14,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-04 17:57:14,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-04 17:57:14,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-04 17:57:14,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-04 17:57:14,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-04 17:57:14,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-04 17:57:14,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-04 17:57:14,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-04 17:57:14,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-04 17:57:14,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-04 17:57:14,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-04 17:57:14,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-04 17:57:14,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-04 17:57:14,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-04 17:57:14,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-04 17:57:14,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-04 17:57:14,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-04 17:57:14,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-04 17:57:14,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-04 17:57:14,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-04 17:57:14,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-04 17:57:14,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-04 17:57:14,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-04 17:57:14,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-04 17:57:14,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-04 17:57:14,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-04 17:57:14,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-04 17:57:14,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-04 17:57:14,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-04 17:57:14,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-04 17:57:14,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-04 17:57:14,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-04 17:57:14,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-04 17:57:14,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-04 17:57:14,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-04 17:57:14,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-04 17:57:14,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-04 17:57:14,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-04 17:57:14,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-04 17:57:14,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-04 17:57:14,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-04 17:57:14,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-04 17:57:14,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-04 17:57:14,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-04 17:57:14,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-04 17:57:14,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-04 17:57:14,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-04 17:57:14,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-04 17:57:14,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-04 17:57:14,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-04 17:57:14,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-04 17:57:14,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-04 17:57:14,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-04 17:57:14,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-04 17:57:14,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-04 17:57:14,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-04 17:57:14,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-04 17:57:14,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-04 17:57:14,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-04 17:57:14,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-04 17:57:14,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-04 17:57:14,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-04 17:57:14,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-04 17:57:14,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-04 17:57:14,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-04 17:57:14,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-04 17:57:14,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-04 17:57:14,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-04 17:57:14,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-04 17:57:14,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-04 17:57:14,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-04 17:57:14,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-04 17:57:14,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-04 17:57:14,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-04 17:57:14,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-04 17:57:14,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-04 17:57:14,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-04 17:57:14,479 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006321005190772531); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0067834882385377025); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006321005190772531); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0045); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-04 17:57:14,479 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-04 17:57:14,479 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.912656556232658E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006321005190772531); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0067834882385377025); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006321005190772531); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0045); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-04 17:57:14,707 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 302 ms to try 5000 different iterations.  Found a solution that moves 230 regions; Going from a computed imbalance of 9.912656556232658E-4 to a new imbalance of 1.9385050782554646E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.025555555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-04 17:57:14,708 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1083809150:83 , srv1557461030:83 , srv652485976:83 , srv1024080925:84 , srv1135866950:84 , srv116656732:84 , srv1201225717:84 , srv1215770786:84 , srv1219518025:84 , srv1289721902:84 , srv133388558:84 , srv1403859699:84 , srv1510910502:84 , srv1532623531:84 , srv155741663:84 , srv161603882:84 , srv1652991268:84 , srv1716698314:84 , srv1727403267:84 , srv1731925965:84 , srv1767900481:84 , srv1780882452:84 , srv1900202198:84 , srv1933428091:84 , srv1951791612:84 , srv2040198677:84 , srv2064678088:84 , srv2096912194:84 , srv2117757312:84 , srv228127437:84 , srv305122922:84 , srv336585116:84 , srv345900100:84 , srv439430690:84 , srv445367394:84 , srv529364755:84 , srv537372672:84 , srv679673696:84 , srv782113315:84 , srv792818731:84 , srv802898905:84 , srv844555959:84 , srv896840150:84 , srv92354283:84 , srv959018295:84 , srv99254213:84 , srv1016373651:85 , srv1053848651:85 , srv1058874041:85 , srv1068812114:85 , srv1125439563:85 , srv1145582238:85 , srv1165410730:85 , srv1181992614:85 , srv1187450016:85 , srv1203442755:85 , srv1222324183:85 , srv1252553261:85 , srv1371416379:85 , srv1394588950:85 , srv1408277952:85 , srv1421319842:85 , srv1430857813:85 , srv1488602806:85 , srv1510213411:85 , srv1532628946:85 , srv1676566076:85 , srv1682106945:85 , srv1732778173:85 , srv1818501412:85 , srv1835986763:85 , srv1911377082:85 , srv1945872937:85 , srv2057786855:85 , srv2139476480:85 , srv214192825:85 , srv242164726:85 , srv299778329:85 , srv339484475:85 , srv362074472:85 , srv372077928:85 , srv401799100:85 , srv490875092:85 , srv504634409:85 , srv540016811:85 , srv733020029:85 , srv763791921:85 , srv787019510:85 , srv96355184:85 , srv970492173:85 , srv975323362:85 , srv987585921:85 , srv996566379:85 , srv1455744920:86 , srv1801236030:86 , srv2027149052:86 , srv404059649:86 , srv730556340:86 , srv886640142:86 , srv1604525141:628 }
2023-04-04 17:57:14,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-04 17:57:14,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1732778173=43, srv1371416379=21, srv2027149052=54, srv490875092=75, srv1652991268=37, srv996566379=99, srv1083809150=5, srv1222324183=17, srv336585116=66, srv1532628946=32, srv155741663=33, srv1731925965=42, srv362074472=69, srv1289721902=19, srv1532623531=31, srv1801236030=46, srv1900202198=49, srv2040198677=55, srv792818731=87, srv537372672=78, srv1394588950=22, srv1215770786=15, srv228127437=62, srv1024080925=1, srv133388558=20, srv1488602806=28, srv787019510=86, srv540016811=79, srv1219518025=16, srv1945872937=52, srv1016373651=0, srv1181992614=11, srv99254213=98, srv1165410730=9, srv1716698314=40, srv679673696=81, srv404059649=72, srv2139476480=60, srv504634409=76, srv445367394=74, srv1676566076=38, srv1201225717=13, srv1135866950=7, srv372077928=70, srv161603882=36, srv733020029=83, srv529364755=77, srv652485976=80, srv1145582238=8, srv1911377082=50, srv2057786855=56, srv1408277952=24, srv896840150=91, srv1835986763=48, srv242164726=63, srv92354283=92, srv1682106945=39, srv959018295=93, srv1951791612=53, srv1403859699=23, srv802898905=88, srv401799100=71, srv1421319842=25, srv299778329=64, srv970492173=95, srv1604525141=35, srv1727403267=41, srv1933428091=51, srv305122922=65, srv345900100=68, srv1455744920=27, srv2117757312=59, srv439430690=73, srv214192825=61, srv1187450016=12, srv763791921=84, srv1430857813=26, srv1510213411=29, srv1557461030=34, srv1068812114=4, srv1053848651=2, srv339484475=67, srv1203442755=14, srv2096912194=58, srv975323362=96, srv1058874041=3, srv844555959=89, srv1125439563=6, srv782113315=85, srv1767900481=44, srv1818501412=47, srv116656732=10, srv1510910502=30, srv96355184=94, srv987585921=97, srv2064678088=57, srv1252553261=18, srv730556340=82, srv1780882452=45, srv886640142=90} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-04 17:57:14,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-04 17:57:14,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-04 17:57:14,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-04 17:57:14,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-04 17:57:14,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-04 17:57:14,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-04 17:57:14,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-04 17:57:14,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-04 17:57:14,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-04 17:57:14,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-04 17:57:14,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-04 17:57:14,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-04 17:57:14,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-04 17:57:14,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-04 17:57:14,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-04 17:57:14,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-04 17:57:14,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-04 17:57:14,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-04 17:57:14,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-04 17:57:14,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-04 17:57:14,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-04 17:57:14,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-04 17:57:14,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-04 17:57:14,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-04 17:57:14,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-04 17:57:14,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-04 17:57:14,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-04 17:57:14,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-04 17:57:14,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-04 17:57:14,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-04 17:57:14,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-04 17:57:14,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-04 17:57:14,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-04 17:57:14,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-04 17:57:14,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-04 17:57:14,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-04 17:57:14,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-04 17:57:14,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-04 17:57:14,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-04 17:57:14,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-04 17:57:14,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-04 17:57:14,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-04 17:57:14,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-04 17:57:14,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-04 17:57:14,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-04 17:57:14,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-04 17:57:14,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-04 17:57:14,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-04 17:57:14,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-04 17:57:14,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-04 17:57:14,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-04 17:57:14,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-04 17:57:14,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-04 17:57:14,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-04 17:57:14,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-04 17:57:14,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-04 17:57:14,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-04 17:57:14,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-04 17:57:14,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-04 17:57:14,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-04 17:57:14,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-04 17:57:14,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-04 17:57:14,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-04 17:57:14,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-04 17:57:14,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-04 17:57:14,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-04 17:57:14,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-04 17:57:14,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-04 17:57:14,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-04 17:57:14,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-04 17:57:14,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-04 17:57:14,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-04 17:57:14,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-04 17:57:14,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-04 17:57:14,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-04 17:57:14,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-04 17:57:14,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-04 17:57:14,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-04 17:57:14,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-04 17:57:14,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-04 17:57:14,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-04 17:57:14,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-04 17:57:14,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-04 17:57:14,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-04 17:57:14,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-04 17:57:14,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-04 17:57:14,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-04 17:57:14,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-04 17:57:14,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-04 17:57:14,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-04 17:57:14,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-04 17:57:14,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-04 17:57:14,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-04 17:57:14,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-04 17:57:14,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-04 17:57:14,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-04 17:57:14,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-04 17:57:14,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-04 17:57:14,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-04 17:57:14,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-04 17:57:14,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-04 17:57:14,812 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.777455636076716E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-04 17:57:14,812 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1083809150:83 , srv1557461030:83 , srv652485976:83 , srv1024080925:84 , srv1135866950:84 , srv116656732:84 , srv1201225717:84 , srv1215770786:84 , srv1219518025:84 , srv1289721902:84 , srv133388558:84 , srv1403859699:84 , srv1510910502:84 , srv1532623531:84 , srv155741663:84 , srv161603882:84 , srv1652991268:84 , srv1716698314:84 , srv1727403267:84 , srv1731925965:84 , srv1767900481:84 , srv1780882452:84 , srv1900202198:84 , srv1933428091:84 , srv1951791612:84 , srv2040198677:84 , srv2064678088:84 , srv2096912194:84 , srv2117757312:84 , srv228127437:84 , srv305122922:84 , srv336585116:84 , srv345900100:84 , srv439430690:84 , srv445367394:84 , srv529364755:84 , srv537372672:84 , srv679673696:84 , srv782113315:84 , srv792818731:84 , srv802898905:84 , srv844555959:84 , srv896840150:84 , srv92354283:84 , srv959018295:84 , srv99254213:84 , srv1016373651:85 , srv1053848651:85 , srv1058874041:85 , srv1068812114:85 , srv1125439563:85 , srv1145582238:85 , srv1165410730:85 , srv1181992614:85 , srv1187450016:85 , srv1203442755:85 , srv1222324183:85 , srv1252553261:85 , srv1371416379:85 , srv1394588950:85 , srv1408277952:85 , srv1421319842:85 , srv1430857813:85 , srv1488602806:85 , srv1510213411:85 , srv1532628946:85 , srv1676566076:85 , srv1682106945:85 , srv1732778173:85 , srv1818501412:85 , srv1835986763:85 , srv1911377082:85 , srv1945872937:85 , srv2057786855:85 , srv2139476480:85 , srv214192825:85 , srv242164726:85 , srv299778329:85 , srv339484475:85 , srv362074472:85 , srv372077928:85 , srv401799100:85 , srv490875092:85 , srv504634409:85 , srv540016811:85 , srv733020029:85 , srv763791921:85 , srv787019510:85 , srv96355184:85 , srv970492173:85 , srv975323362:85 , srv987585921:85 , srv996566379:85 , srv1455744920:86 , srv1801236030:86 , srv2027149052:86 , srv404059649:86 , srv730556340:86 , srv886640142:86 , srv1604525141:628 }
2023-04-04 17:57:14,839 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=291 (was 282) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 171), AvailableMemoryMB=7646 (was 8124)
2023-04-04 17:57:14,846 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=291, ProcessCount=169, AvailableMemoryMB=7645
2023-04-04 17:57:14,847 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-04 17:57:14,847 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-04-04 17:57:14,856 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1109403979:84 , srv116563151:84 , srv1178235405:84 , srv1208288321:84 , srv1235615635:84 , srv1276670799:84 , srv1304378516:84 , srv1370947614:84 , srv1547547276:84 , srv1577560411:84 , srv1695703910:84 , srv175953028:84 , srv1792359572:84 , srv1892298579:84 , srv1971796458:84 , srv1979863015:84 , srv1994802377:84 , srv2025324919:84 , srv205190729:84 , srv2096859102:84 , srv471087953:84 , srv526756682:84 , srv543634150:84 , srv550023738:84 , srv63235208:84 , srv857241358:84 , srv881279593:84 , srv967221173:84 , srv994783535:84 , srv270382761:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-04-04 17:57:14,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-04 17:57:14,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv205190729=18, srv1547547276=8, srv1792359572=12, srv967221173=28, srv1208288321=3, srv270382761=20, srv63235208=25, srv994783535=29, srv1971796458=14, srv1994802377=16, srv1892298579=13, srv2025324919=17, srv471087953=21, srv1695703910=10, srv1304378516=6, srv1370947614=7, srv857241358=26, srv1109403979=0, srv1178235405=2, srv1276670799=5, srv175953028=11, srv116563151=1, srv543634150=23, srv881279593=27, srv526756682=22, srv550023738=24, srv1979863015=15, srv1577560411=9, srv1235615635=4, srv2096859102=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-04 17:57:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-04 17:57:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-04 17:57:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-04 17:57:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-04 17:57:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-04 17:57:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-04 17:57:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-04 17:57:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-04 17:57:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-04 17:57:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-04 17:57:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-04 17:57:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-04 17:57:14,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-04 17:57:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-04 17:57:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-04 17:57:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-04 17:57:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-04 17:57:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-04 17:57:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-04 17:57:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-04 17:57:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-04 17:57:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-04 17:57:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-04 17:57:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-04 17:57:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-04 17:57:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-04 17:57:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-04 17:57:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-04 17:57:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-04 17:57:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-04 17:57:14,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-04 17:57:14,883 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1351506456513658, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-04 17:57:14,883 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912618898535652, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1351506456513658, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-04-04 17:59:14,873 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9749786 different iterations.  Found a solution that moves 1837 regions; Going from a computed imbalance of 0.9912618898535652 to a new imbalance of 4.2876496417733574E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6803703703703704, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-04 17:59:14,878 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1109403979:90 , srv116563151:90 , srv1178235405:90 , srv1208288321:90 , srv1235615635:90 , srv1276670799:90 , srv1304378516:90 , srv1370947614:90 , srv1547547276:90 , srv1577560411:90 , srv1695703910:90 , srv175953028:90 , srv1792359572:90 , srv1892298579:90 , srv1971796458:90 , srv1979863015:90 , srv1994802377:90 , srv2025324919:90 , srv205190729:90 , srv2096859102:90 , srv270382761:90 , srv471087953:90 , srv526756682:90 , srv543634150:90 , srv550023738:90 , srv63235208:90 , srv857241358:90 , srv881279593:90 , srv967221173:90 , srv994783535:90 }
2023-04-04 17:59:14,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-04 17:59:14,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv205190729=18, srv1547547276=8, srv1792359572=12, srv967221173=28, srv1208288321=3, srv270382761=20, srv63235208=25, srv994783535=29, srv1971796458=14, srv1994802377=16, srv1892298579=13, srv2025324919=17, srv471087953=21, srv1695703910=10, srv1304378516=6, srv1370947614=7, srv857241358=26, srv1109403979=0, srv1178235405=2, srv1276670799=5, srv175953028=11, srv116563151=1, srv543634150=23, srv881279593=27, srv526756682=22, srv550023738=24, srv1979863015=15, srv1577560411=9, srv1235615635=4, srv2096859102=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-04 17:59:14,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-04 17:59:14,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-04 17:59:14,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-04 17:59:14,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-04 17:59:14,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-04 17:59:14,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-04 17:59:14,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-04 17:59:14,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-04 17:59:14,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-04 17:59:14,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-04 17:59:14,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-04 17:59:14,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-04 17:59:14,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-04 17:59:14,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-04 17:59:14,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-04 17:59:14,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-04 17:59:14,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-04 17:59:14,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-04 17:59:14,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-04 17:59:14,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-04 17:59:14,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-04 17:59:14,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-04 17:59:14,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-04 17:59:14,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-04 17:59:14,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-04 17:59:14,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-04 17:59:14,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-04 17:59:14,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-04 17:59:14,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-04 17:59:14,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-04 17:59:14,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-04 17:59:14,893 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-04 17:59:14,894 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1109403979:90 , srv116563151:90 , srv1178235405:90 , srv1208288321:90 , srv1235615635:90 , srv1276670799:90 , srv1304378516:90 , srv1370947614:90 , srv1547547276:90 , srv1577560411:90 , srv1695703910:90 , srv175953028:90 , srv1792359572:90 , srv1892298579:90 , srv1971796458:90 , srv1979863015:90 , srv1994802377:90 , srv2025324919:90 , srv205190729:90 , srv2096859102:90 , srv270382761:90 , srv471087953:90 , srv526756682:90 , srv543634150:90 , srv550023738:90 , srv63235208:90 , srv857241358:90 , srv881279593:90 , srv967221173:90 , srv994783535:90 }
2023-04-04 17:59:14,904 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=269 (was 291), ProcessCount=169 (was 169), AvailableMemoryMB=7466 (was 7645)