2023-09-26 01:58:44,400 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-09-26 01:58:44,628 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-09-26 01:58:44,724 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-09-26 01:58:44,777 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-26 01:58:44,783 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-09-26 01:58:44,794 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=135, ProcessCount=166, AvailableMemoryMB=8273
2023-09-26 01:58:44,796 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-26 01:58:44,796 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-09-26 01:58:44,953 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1049834700:84 , srv1101140186:84 , srv1146929175:84 , srv1196638697:84 , srv1201213592:84 , srv1209517805:84 , srv1240882252:84 , srv1302358587:84 , srv1310157919:84 , srv1318828499:84 , srv1328114476:84 , srv1346272732:84 , srv134833938:84 , srv1414104133:84 , srv1420122369:84 , srv1458397652:84 , srv1466597012:84 , srv1477154116:84 , srv150177733:84 , srv1578969763:84 , srv1606002261:84 , srv165699305:84 , srv1663750:84 , srv1682413768:84 , srv1698542698:84 , srv17659757:84 , srv1784902581:84 , srv178715911:84 , srv1820937409:84 , srv1828513910:84 , srv1829394076:84 , srv1829692473:84 , srv1842556185:84 , srv1874597875:84 , srv1878564055:84 , srv1951490590:84 , srv1958146009:84 , srv1960447602:84 , srv196255433:84 , srv1970290355:84 , srv2005386537:84 , srv2014494699:84 , srv2054099354:84 , srv2054240404:84 , srv2077003024:84 , srv2089146179:84 , srv2120395334:84 , srv223102748:84 , srv227436634:84 , srv23138020:84 , srv237913704:84 , srv239831769:84 , srv240766402:84 , srv24660948:84 , srv247933306:84 , srv266409999:84 , srv320201703:84 , srv339598953:84 , srv364135130:84 , srv374152917:84 , srv37877662:84 , srv380009089:84 , srv388969810:84 , srv405759472:84 , srv423123302:84 , srv456111782:84 , srv461538518:84 , srv502482839:84 , srv538615922:84 , srv54508703:84 , srv548866751:84 , srv550557679:84 , srv563594673:84 , srv579192810:84 , srv584677683:84 , srv590608334:84 , srv593438444:84 , srv595272635:84 , srv615639059:84 , srv627857943:84 , srv632749287:84 , srv643927896:84 , srv645862840:84 , srv70239008:84 , srv703125158:84 , srv73850172:84 , srv74181935:84 , srv752767864:84 , srv779999860:84 , srv819333271:84 , srv819866202:84 , srv820742007:84 , srv82414453:84 , srv849973048:84 , srv850074997:84 , srv898419999:84 , srv910293841:84 , srv932835779:84 , srv967592295:84 , srv534944238:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-09-26 01:58:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-26 01:58:44,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv752767864=88, srv1477154116=17, srv1878564055=34, srv423123302=64, srv2014494699=41, srv643927896=82, srv1146929175=2, srv1310157919=8, srv388969810=62, srv178715911=27, srv2120395334=46, srv73850172=86, srv2054240404=43, srv24660948=53, srv563594673=73, srv850074997=95, srv134833938=12, srv1960447602=37, srv82414453=93, srv227436634=48, srv538615922=69, srv1828513910=29, srv534944238=68, srv54508703=70, srv1829692473=31, srv456111782=65, srv1784902581=26, srv1458397652=15, srv461538518=66, srv37877662=60, srv1328114476=10, srv595272635=78, srv1346272732=11, srv380009089=61, srv165699305=21, srv405759472=63, srv1209517805=5, srv1682413768=23, srv1842556185=32, srv1201213592=4, srv1820937409=28, srv820742007=92, srv1466597012=16, srv819866202=91, srv1698542698=24, srv593438444=77, srv703125158=85, srv967592295=99, srv849973048=94, srv2054099354=42, srv584677683=75, srv240766402=52, srv23138020=49, srv1951490590=35, srv1420122369=14, srv1606002261=20, srv2089146179=45, srv502482839=67, srv1958146009=36, srv239831769=51, srv320201703=56, srv590608334=76, srv364135130=58, srv1101140186=1, srv1318828499=9, srv1874597875=33, srv2005386537=40, srv17659757=25, srv579192810=74, srv74181935=87, srv1302358587=7, srv615639059=79, srv898419999=96, srv2077003024=44, srv237913704=50, srv1240882252=6, srv1578969763=19, srv1196638697=3, srv932835779=98, srv223102748=47, srv1829394076=30, srv1663750=22, srv645862840=83, srv196255433=38, srv548866751=71, srv632749287=81, srv779999860=89, srv819333271=90, srv910293841=97, srv1049834700=0, srv247933306=54, srv1970290355=39, srv339598953=57, srv374152917=59, srv266409999=55, srv550557679=72, srv70239008=84, srv1414104133=13, srv627857943=80, srv150177733=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-26 01:58:45,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-26 01:58:45,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-26 01:58:45,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-26 01:58:45,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-26 01:58:45,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-26 01:58:45,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-26 01:58:45,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-26 01:58:45,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-26 01:58:45,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-26 01:58:45,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-26 01:58:45,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-26 01:58:45,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-26 01:58:45,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-26 01:58:45,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-26 01:58:45,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-26 01:58:45,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-26 01:58:45,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-26 01:58:45,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-26 01:58:45,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-26 01:58:45,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-26 01:58:45,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-26 01:58:45,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-26 01:58:45,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-26 01:58:45,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-26 01:58:45,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-26 01:58:45,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-26 01:58:45,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-26 01:58:45,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-26 01:58:45,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-26 01:58:45,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-26 01:58:45,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-26 01:58:45,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-26 01:58:45,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-26 01:58:45,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-26 01:58:45,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-26 01:58:45,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-26 01:58:45,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-26 01:58:45,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-26 01:58:45,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-26 01:58:45,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-26 01:58:45,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-26 01:58:45,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-26 01:58:45,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-26 01:58:45,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-26 01:58:45,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-26 01:58:45,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-26 01:58:45,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-26 01:58:45,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-26 01:58:45,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-26 01:58:45,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-26 01:58:45,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-26 01:58:45,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-26 01:58:45,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-26 01:58:45,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-26 01:58:45,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-26 01:58:45,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-26 01:58:45,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-26 01:58:45,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-26 01:58:45,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-26 01:58:45,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-26 01:58:45,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-26 01:58:45,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-26 01:58:45,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-26 01:58:45,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-26 01:58:45,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-26 01:58:45,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-26 01:58:45,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-26 01:58:45,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-26 01:58:45,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-26 01:58:45,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-26 01:58:45,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-26 01:58:45,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-26 01:58:45,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-26 01:58:45,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-26 01:58:45,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-26 01:58:45,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-26 01:58:45,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-26 01:58:45,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-26 01:58:45,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-26 01:58:45,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-26 01:58:45,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-26 01:58:45,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-26 01:58:45,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-26 01:58:45,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-26 01:58:45,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-26 01:58:45,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-26 01:58:45,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-26 01:58:45,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-26 01:58:45,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-26 01:58:45,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-26 01:58:45,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-26 01:58:45,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-26 01:58:45,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-26 01:58:45,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-26 01:58:45,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-26 01:58:45,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-26 01:58:45,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-26 01:58:45,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-26 01:58:45,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-26 01:58:45,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-26 01:58:45,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-26 01:58:45,171 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-26 01:58:45,171 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-26 01:58:45,171 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-26 01:58:45,432 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 366 ms to try 5000 different iterations.  Found a solution that moves 4718 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18016702292001402. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03905128829623563, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05570082018469778, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5242222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03905128829623563, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17558333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24016666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-26 01:58:45,453 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv643927896:82 , srv1049834700:83 , srv1101140186:83 , srv165699305:83 , srv178715911:83 , srv1829394076:83 , srv1878564055:83 , srv2077003024:83 , srv380009089:83 , srv405759472:83 , srv461538518:83 , srv584677683:83 , srv73850172:83 , srv967592295:83 , srv17659757:84 , srv1784902581:84 , srv247933306:84 , srv423123302:84 , srv74181935:84 , srv820742007:84 , srv849973048:84 , srv850074997:84 , srv1146929175:85 , srv1196638697:85 , srv150177733:85 , srv1829692473:85 , srv1951490590:85 , srv2054240404:85 , srv239831769:85 , srv364135130:85 , srv456111782:85 , srv590608334:85 , srv595272635:85 , srv632749287:85 , srv910293841:85 , srv1328114476:86 , srv134833938:86 , srv1477154116:86 , srv1578969763:86 , srv1682413768:86 , srv2014494699:86 , srv2089146179:86 , srv2120395334:86 , srv227436634:86 , srv23138020:86 , srv240766402:86 , srv266409999:86 , srv339598953:86 , srv548866751:86 , srv550557679:86 , srv563594673:86 , srv579192810:86 , srv615639059:86 , srv70239008:86 , srv1201213592:87 , srv1346272732:87 , srv1420122369:87 , srv1698542698:87 , srv1820937409:87 , srv1828513910:87 , srv1958146009:87 , srv1960447602:87 , srv1970290355:87 , srv223102748:87 , srv237913704:87 , srv37877662:87 , srv627857943:87 , srv1209517805:88 , srv1302358587:88 , srv1310157919:88 , srv1663750:88 , srv1842556185:88 , srv196255433:88 , srv2005386537:88 , srv2054099354:88 , srv24660948:88 , srv320201703:88 , srv374152917:88 , srv388969810:88 , srv645862840:88 , srv752767864:88 , srv819333271:88 , srv819866202:88 , srv1318828499:89 , srv1606002261:89 , srv1874597875:89 , srv538615922:89 , srv593438444:89 , srv703125158:89 , srv932835779:89 , srv1414104133:90 , srv898419999:90 , srv1240882252:91 , srv54508703:91 , srv1458397652:92 , srv502482839:92 , srv779999860:92 , srv82414453:95 , srv1466597012:96 , srv534944238:437 }
2023-09-26 01:58:45,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-26 01:58:45,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv752767864=88, srv1477154116=17, srv1878564055=34, srv423123302=64, srv2014494699=41, srv643927896=82, srv1146929175=2, srv1310157919=8, srv388969810=62, srv178715911=27, srv2120395334=46, srv73850172=86, srv2054240404=43, srv24660948=53, srv563594673=73, srv850074997=95, srv134833938=12, srv1960447602=37, srv82414453=93, srv227436634=48, srv538615922=69, srv1828513910=29, srv534944238=68, srv54508703=70, srv1829692473=31, srv456111782=65, srv1784902581=26, srv1458397652=15, srv461538518=66, srv37877662=60, srv1328114476=10, srv595272635=78, srv1346272732=11, srv380009089=61, srv165699305=21, srv405759472=63, srv1209517805=5, srv1682413768=23, srv1842556185=32, srv1201213592=4, srv1820937409=28, srv820742007=92, srv1466597012=16, srv819866202=91, srv1698542698=24, srv593438444=77, srv703125158=85, srv967592295=99, srv849973048=94, srv2054099354=42, srv584677683=75, srv240766402=52, srv23138020=49, srv1951490590=35, srv1420122369=14, srv1606002261=20, srv2089146179=45, srv502482839=67, srv1958146009=36, srv239831769=51, srv320201703=56, srv590608334=76, srv364135130=58, srv1101140186=1, srv1318828499=9, srv1874597875=33, srv2005386537=40, srv17659757=25, srv579192810=74, srv74181935=87, srv1302358587=7, srv615639059=79, srv898419999=96, srv2077003024=44, srv237913704=50, srv1240882252=6, srv1578969763=19, srv1196638697=3, srv932835779=98, srv223102748=47, srv1829394076=30, srv1663750=22, srv645862840=83, srv196255433=38, srv548866751=71, srv632749287=81, srv779999860=89, srv819333271=90, srv910293841=97, srv1049834700=0, srv247933306=54, srv1970290355=39, srv339598953=57, srv374152917=59, srv266409999=55, srv550557679=72, srv70239008=84, srv1414104133=13, srv627857943=80, srv150177733=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-26 01:58:45,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-26 01:58:45,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-26 01:58:45,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-26 01:58:45,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-26 01:58:45,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-26 01:58:45,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-26 01:58:45,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-26 01:58:45,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-26 01:58:45,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-26 01:58:45,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-26 01:58:45,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-26 01:58:45,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-26 01:58:45,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-26 01:58:45,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-26 01:58:45,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-26 01:58:45,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-26 01:58:45,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-26 01:58:45,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-26 01:58:45,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-26 01:58:45,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-26 01:58:45,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-26 01:58:45,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-26 01:58:45,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-26 01:58:45,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-26 01:58:45,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-26 01:58:45,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-26 01:58:45,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-26 01:58:45,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-26 01:58:45,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-26 01:58:45,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-26 01:58:45,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-26 01:58:45,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-26 01:58:45,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-26 01:58:45,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-26 01:58:45,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-26 01:58:45,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-26 01:58:45,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-26 01:58:45,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-26 01:58:45,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-26 01:58:45,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-26 01:58:45,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-26 01:58:45,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-26 01:58:45,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-26 01:58:45,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-26 01:58:45,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-26 01:58:45,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-26 01:58:45,479 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-26 01:58:45,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-26 01:58:45,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-26 01:58:45,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-26 01:58:45,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-26 01:58:45,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-26 01:58:45,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-26 01:58:45,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-26 01:58:45,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-26 01:58:45,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-26 01:58:45,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-26 01:58:45,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-26 01:58:45,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-26 01:58:45,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-26 01:58:45,480 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-26 01:58:45,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-26 01:58:45,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-26 01:58:45,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-26 01:58:45,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-26 01:58:45,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-26 01:58:45,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-26 01:58:45,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-26 01:58:45,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-26 01:58:45,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-26 01:58:45,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-26 01:58:45,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-26 01:58:45,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-26 01:58:45,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-26 01:58:45,481 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-26 01:58:45,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-26 01:58:45,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-26 01:58:45,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-26 01:58:45,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-26 01:58:45,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-26 01:58:45,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-26 01:58:45,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-26 01:58:45,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-26 01:58:45,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-26 01:58:45,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-26 01:58:45,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-26 01:58:45,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-26 01:58:45,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-26 01:58:45,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-26 01:58:45,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-26 01:58:45,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-26 01:58:45,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-26 01:58:45,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-26 01:58:45,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-26 01:58:45,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-26 01:58:45,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-26 01:58:45,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-26 01:58:45,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-26 01:58:45,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-26 01:58:45,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-26 01:58:45,483 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-26 01:58:45,590 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03905128829623563, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05570082018469778, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03905128829623563, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17558333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24016666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-26 01:58:45,591 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-26 01:58:45,591 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1801339867779184, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03905128829623563, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05570082018469778, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03905128829623563, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17558333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24016666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-26 01:58:45,826 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 321 ms to try 5000 different iterations.  Found a solution that moves 2609 regions; Going from a computed imbalance of 0.1801339867779184 to a new imbalance of 0.01402160734497889. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01935485211868658); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03352135523833412, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28988888888888886, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.01935485211868658); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043666666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-26 01:58:45,835 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1466597012:76 , srv779999860:80 , srv82414453:80 , srv1240882252:81 , srv1318828499:81 , srv1458397652:81 , srv538615922:81 , srv1874597875:82 , srv502482839:82 , srv54508703:82 , srv645862840:82 , srv1209517805:83 , srv1414104133:83 , srv1663750:83 , srv703125158:83 , srv819333271:83 , srv1310157919:84 , srv1820937409:84 , srv1828513910:84 , srv1842556185:84 , srv1960447602:84 , srv2014494699:84 , srv2054099354:84 , srv320201703:84 , srv932835779:84 , srv1201213592:85 , srv1346272732:85 , srv1420122369:85 , srv1477154116:85 , srv1682413768:85 , srv1958146009:85 , srv196255433:85 , srv2005386537:85 , srv2089146179:85 , srv2120395334:85 , srv23138020:85 , srv237913704:85 , srv24660948:85 , srv548866751:85 , srv593438444:85 , srv627857943:85 , srv752767864:85 , srv819866202:85 , srv898419999:85 , srv1146929175:86 , srv1578969763:86 , srv1606002261:86 , srv1698542698:86 , srv1970290355:86 , srv223102748:86 , srv239831769:86 , srv266409999:86 , srv364135130:86 , srv374152917:86 , srv37877662:86 , srv456111782:86 , srv550557679:86 , srv563594673:86 , srv615639059:86 , srv1196638697:87 , srv1302358587:87 , srv1328114476:87 , srv134833938:87 , srv17659757:87 , srv1784902581:87 , srv1829692473:87 , srv2077003024:87 , srv227436634:87 , srv339598953:87 , srv388969810:87 , srv423123302:87 , srv579192810:87 , srv590608334:87 , srv632749287:87 , srv820742007:87 , srv1049834700:88 , srv150177733:88 , srv1951490590:88 , srv2054240404:88 , srv240766402:88 , srv380009089:88 , srv405759472:88 , srv461538518:88 , srv595272635:88 , srv70239008:88 , srv74181935:88 , srv849973048:88 , srv850074997:88 , srv910293841:88 , srv1101140186:89 , srv165699305:89 , srv1829394076:89 , srv178715911:90 , srv1878564055:90 , srv967592295:90 , srv584677683:91 , srv73850172:91 , srv247933306:92 , srv643927896:92 , srv534944238:509 }
2023-09-26 01:58:45,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-26 01:58:45,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv752767864=88, srv1477154116=17, srv1878564055=34, srv423123302=64, srv2014494699=41, srv643927896=82, srv1146929175=2, srv1310157919=8, srv388969810=62, srv178715911=27, srv2120395334=46, srv73850172=86, srv2054240404=43, srv24660948=53, srv563594673=73, srv850074997=95, srv134833938=12, srv1960447602=37, srv82414453=93, srv227436634=48, srv538615922=69, srv1828513910=29, srv534944238=68, srv54508703=70, srv1829692473=31, srv456111782=65, srv1784902581=26, srv1458397652=15, srv461538518=66, srv37877662=60, srv1328114476=10, srv595272635=78, srv1346272732=11, srv380009089=61, srv165699305=21, srv405759472=63, srv1209517805=5, srv1682413768=23, srv1842556185=32, srv1201213592=4, srv1820937409=28, srv820742007=92, srv1466597012=16, srv819866202=91, srv1698542698=24, srv593438444=77, srv703125158=85, srv967592295=99, srv849973048=94, srv2054099354=42, srv584677683=75, srv240766402=52, srv23138020=49, srv1951490590=35, srv1420122369=14, srv1606002261=20, srv2089146179=45, srv502482839=67, srv1958146009=36, srv239831769=51, srv320201703=56, srv590608334=76, srv364135130=58, srv1101140186=1, srv1318828499=9, srv1874597875=33, srv2005386537=40, srv17659757=25, srv579192810=74, srv74181935=87, srv1302358587=7, srv615639059=79, srv898419999=96, srv2077003024=44, srv237913704=50, srv1240882252=6, srv1578969763=19, srv1196638697=3, srv932835779=98, srv223102748=47, srv1829394076=30, srv1663750=22, srv645862840=83, srv196255433=38, srv548866751=71, srv632749287=81, srv779999860=89, srv819333271=90, srv910293841=97, srv1049834700=0, srv247933306=54, srv1970290355=39, srv339598953=57, srv374152917=59, srv266409999=55, srv550557679=72, srv70239008=84, srv1414104133=13, srv627857943=80, srv150177733=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-26 01:58:45,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-26 01:58:45,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-26 01:58:45,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-26 01:58:45,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-26 01:58:45,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-26 01:58:45,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-26 01:58:45,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-26 01:58:45,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-26 01:58:45,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-26 01:58:45,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-26 01:58:45,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-26 01:58:45,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-26 01:58:45,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-26 01:58:45,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-26 01:58:45,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-26 01:58:45,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-26 01:58:45,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-26 01:58:45,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-26 01:58:45,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-26 01:58:45,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-26 01:58:45,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-26 01:58:45,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-26 01:58:45,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-26 01:58:45,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-26 01:58:45,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-26 01:58:45,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-26 01:58:45,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-26 01:58:45,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-26 01:58:45,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-26 01:58:45,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-26 01:58:45,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-26 01:58:45,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-26 01:58:45,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-26 01:58:45,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-26 01:58:45,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-26 01:58:45,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-26 01:58:45,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-26 01:58:45,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-26 01:58:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-26 01:58:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-26 01:58:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-26 01:58:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-26 01:58:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-26 01:58:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-26 01:58:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-26 01:58:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-26 01:58:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-26 01:58:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-26 01:58:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-26 01:58:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-26 01:58:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-26 01:58:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-26 01:58:45,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-26 01:58:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-26 01:58:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-26 01:58:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-26 01:58:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-26 01:58:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-26 01:58:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-26 01:58:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-26 01:58:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-26 01:58:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-26 01:58:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-26 01:58:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-26 01:58:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-26 01:58:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-26 01:58:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-26 01:58:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-26 01:58:45,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-26 01:58:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-26 01:58:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-26 01:58:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-26 01:58:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-26 01:58:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-26 01:58:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-26 01:58:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-26 01:58:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-26 01:58:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-26 01:58:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-26 01:58:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-26 01:58:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-26 01:58:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-26 01:58:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-26 01:58:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-26 01:58:45,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-26 01:58:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-26 01:58:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-26 01:58:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-26 01:58:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-26 01:58:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-26 01:58:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-26 01:58:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-26 01:58:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-26 01:58:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-26 01:58:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-26 01:58:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-26 01:58:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-26 01:58:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-26 01:58:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-26 01:58:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-26 01:58:45,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-26 01:58:45,956 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01935485211868658); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03352135523833412, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.01935485211868658); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043666666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-26 01:58:45,956 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-26 01:58:45,956 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014003338736516092, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01935485211868658); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03352135523833412, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.01935485211868658); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010916666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043666666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-26 01:58:46,170 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 301 ms to try 5000 different iterations.  Found a solution that moves 993 regions; Going from a computed imbalance of 0.014003338736516092 to a new imbalance of 8.021803585628029E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004112157478995831); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007262900206968707); MoveCostFunction : (multiplier=7.0, imbalance=0.11033333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004112157478995831); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-26 01:58:46,173 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1302358587:82 , srv247933306:82 , srv388969810:82 , srv898419999:82 , srv1606002261:83 , srv240766402:83 , srv374152917:83 , srv502482839:83 , srv584677683:83 , srv1328114476:84 , srv134833938:84 , srv150177733:84 , srv1878564055:84 , srv1970290355:84 , srv2005386537:84 , srv2054240404:84 , srv223102748:84 , srv227436634:84 , srv593438444:84 , srv643927896:84 , srv70239008:84 , srv73850172:84 , srv752767864:84 , srv82414453:84 , srv967592295:84 , srv1196638697:85 , srv1310157919:85 , srv1414104133:85 , srv1458397652:85 , srv1466597012:85 , srv165699305:85 , srv1698542698:85 , srv178715911:85 , srv1829394076:85 , srv1829692473:85 , srv1951490590:85 , srv2054099354:85 , srv266409999:85 , srv37877662:85 , srv54508703:85 , srv548866751:85 , srv563594673:85 , srv595272635:85 , srv615639059:85 , srv850074997:85 , srv910293841:85 , srv932835779:85 , srv1049834700:86 , srv1146929175:86 , srv1201213592:86 , srv1240882252:86 , srv1346272732:86 , srv1420122369:86 , srv1477154116:86 , srv1578969763:86 , srv1663750:86 , srv1682413768:86 , srv1820937409:86 , srv1842556185:86 , srv1874597875:86 , srv1958146009:86 , srv1960447602:86 , srv196255433:86 , srv2120395334:86 , srv23138020:86 , srv24660948:86 , srv320201703:86 , srv339598953:86 , srv405759472:86 , srv423123302:86 , srv550557679:86 , srv579192810:86 , srv627857943:86 , srv632749287:86 , srv703125158:86 , srv779999860:86 , srv819333271:86 , srv819866202:86 , srv820742007:86 , srv849973048:86 , srv1101140186:87 , srv1209517805:87 , srv17659757:87 , srv1784902581:87 , srv1828513910:87 , srv2077003024:87 , srv2089146179:87 , srv237913704:87 , srv239831769:87 , srv364135130:87 , srv380009089:87 , srv456111782:87 , srv461538518:87 , srv590608334:87 , srv645862840:87 , srv74181935:87 , srv1318828499:88 , srv538615922:88 , srv2014494699:89 , srv534944238:548 }
2023-09-26 01:58:46,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-26 01:58:46,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv752767864=88, srv1477154116=17, srv1878564055=34, srv423123302=64, srv2014494699=41, srv643927896=82, srv1146929175=2, srv1310157919=8, srv388969810=62, srv178715911=27, srv2120395334=46, srv73850172=86, srv2054240404=43, srv24660948=53, srv563594673=73, srv850074997=95, srv134833938=12, srv1960447602=37, srv82414453=93, srv227436634=48, srv538615922=69, srv1828513910=29, srv534944238=68, srv54508703=70, srv1829692473=31, srv456111782=65, srv1784902581=26, srv1458397652=15, srv461538518=66, srv37877662=60, srv1328114476=10, srv595272635=78, srv1346272732=11, srv380009089=61, srv165699305=21, srv405759472=63, srv1209517805=5, srv1682413768=23, srv1842556185=32, srv1201213592=4, srv1820937409=28, srv820742007=92, srv1466597012=16, srv819866202=91, srv1698542698=24, srv593438444=77, srv703125158=85, srv967592295=99, srv849973048=94, srv2054099354=42, srv584677683=75, srv240766402=52, srv23138020=49, srv1951490590=35, srv1420122369=14, srv1606002261=20, srv2089146179=45, srv502482839=67, srv1958146009=36, srv239831769=51, srv320201703=56, srv590608334=76, srv364135130=58, srv1101140186=1, srv1318828499=9, srv1874597875=33, srv2005386537=40, srv17659757=25, srv579192810=74, srv74181935=87, srv1302358587=7, srv615639059=79, srv898419999=96, srv2077003024=44, srv237913704=50, srv1240882252=6, srv1578969763=19, srv1196638697=3, srv932835779=98, srv223102748=47, srv1829394076=30, srv1663750=22, srv645862840=83, srv196255433=38, srv548866751=71, srv632749287=81, srv779999860=89, srv819333271=90, srv910293841=97, srv1049834700=0, srv247933306=54, srv1970290355=39, srv339598953=57, srv374152917=59, srv266409999=55, srv550557679=72, srv70239008=84, srv1414104133=13, srv627857943=80, srv150177733=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-26 01:58:46,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-26 01:58:46,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-26 01:58:46,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-26 01:58:46,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-26 01:58:46,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-26 01:58:46,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-26 01:58:46,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-26 01:58:46,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-26 01:58:46,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-26 01:58:46,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-26 01:58:46,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-26 01:58:46,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-26 01:58:46,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-26 01:58:46,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-26 01:58:46,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-26 01:58:46,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-26 01:58:46,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-26 01:58:46,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-26 01:58:46,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-26 01:58:46,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-26 01:58:46,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-26 01:58:46,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-26 01:58:46,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-26 01:58:46,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-26 01:58:46,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-26 01:58:46,205 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-26 01:58:46,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-26 01:58:46,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-26 01:58:46,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-26 01:58:46,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-26 01:58:46,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-26 01:58:46,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-26 01:58:46,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-26 01:58:46,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-26 01:58:46,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-26 01:58:46,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-26 01:58:46,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-26 01:58:46,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-26 01:58:46,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-26 01:58:46,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-26 01:58:46,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-26 01:58:46,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-26 01:58:46,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-26 01:58:46,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-26 01:58:46,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-26 01:58:46,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-26 01:58:46,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-26 01:58:46,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-26 01:58:46,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-26 01:58:46,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-26 01:58:46,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-26 01:58:46,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-26 01:58:46,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-26 01:58:46,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-26 01:58:46,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-26 01:58:46,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-26 01:58:46,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-26 01:58:46,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-26 01:58:46,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-26 01:58:46,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-26 01:58:46,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-26 01:58:46,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-26 01:58:46,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-26 01:58:46,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-26 01:58:46,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-26 01:58:46,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-26 01:58:46,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-26 01:58:46,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-26 01:58:46,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-26 01:58:46,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-26 01:58:46,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-26 01:58:46,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-26 01:58:46,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-26 01:58:46,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-26 01:58:46,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-26 01:58:46,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-26 01:58:46,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-26 01:58:46,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-26 01:58:46,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-26 01:58:46,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-26 01:58:46,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-26 01:58:46,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-26 01:58:46,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-26 01:58:46,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-26 01:58:46,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-26 01:58:46,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-26 01:58:46,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-26 01:58:46,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-26 01:58:46,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-26 01:58:46,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-26 01:58:46,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-26 01:58:46,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-26 01:58:46,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-26 01:58:46,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-26 01:58:46,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-26 01:58:46,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-26 01:58:46,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-26 01:58:46,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-26 01:58:46,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-26 01:58:46,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-26 01:58:46,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-26 01:58:46,331 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004112157478995831); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007262900206968707); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004112157478995831); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-26 01:58:46,331 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-26 01:58:46,331 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.952272239504769E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004112157478995831); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007262900206968707); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004112157478995831); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-26 01:58:46,519 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 290 ms to try 5000 different iterations.  Found a solution that moves 191 regions; Going from a computed imbalance of 7.952272239504769E-4 to a new imbalance of 1.0423139049070238E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.021222222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-26 01:58:46,520 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv237913704:82 , srv1101140186:83 , srv1958146009:83 , srv196255433:83 , srv2014494699:83 , srv24660948:83 , srv339598953:83 , srv579192810:83 , srv590608334:83 , srv74181935:83 , srv819866202:83 , srv82414453:83 , srv1049834700:84 , srv1201213592:84 , srv1240882252:84 , srv1318828499:84 , srv1414104133:84 , srv1420122369:84 , srv1458397652:84 , srv1578969763:84 , srv1606002261:84 , srv1698542698:84 , srv17659757:84 , srv1784902581:84 , srv178715911:84 , srv1820937409:84 , srv1828513910:84 , srv1829692473:84 , srv1842556185:84 , srv1951490590:84 , srv2089146179:84 , srv239831769:84 , srv320201703:84 , srv364135130:84 , srv37877662:84 , srv380009089:84 , srv456111782:84 , srv538615922:84 , srv54508703:84 , srv593438444:84 , srv595272635:84 , srv627857943:84 , srv632749287:84 , srv643927896:84 , srv70239008:84 , srv703125158:84 , srv73850172:84 , srv752767864:84 , srv849973048:84 , srv898419999:84 , srv910293841:84 , srv932835779:84 , srv1196638697:85 , srv1209517805:85 , srv1302358587:85 , srv1310157919:85 , srv1328114476:85 , srv1346272732:85 , srv134833938:85 , srv1466597012:85 , srv1477154116:85 , srv150177733:85 , srv165699305:85 , srv1663750:85 , srv1682413768:85 , srv1829394076:85 , srv1874597875:85 , srv1878564055:85 , srv1960447602:85 , srv1970290355:85 , srv2005386537:85 , srv2054099354:85 , srv2054240404:85 , srv2077003024:85 , srv2120395334:85 , srv223102748:85 , srv227436634:85 , srv23138020:85 , srv240766402:85 , srv247933306:85 , srv266409999:85 , srv374152917:85 , srv388969810:85 , srv405759472:85 , srv423123302:85 , srv461538518:85 , srv502482839:85 , srv550557679:85 , srv563594673:85 , srv584677683:85 , srv615639059:85 , srv645862840:85 , srv779999860:85 , srv819333271:85 , srv820742007:85 , srv850074997:85 , srv967592295:85 , srv1146929175:86 , srv548866751:86 , srv534944238:648 }
2023-09-26 01:58:46,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-26 01:58:46,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv752767864=88, srv1477154116=17, srv1878564055=34, srv423123302=64, srv2014494699=41, srv643927896=82, srv1146929175=2, srv1310157919=8, srv388969810=62, srv178715911=27, srv2120395334=46, srv73850172=86, srv2054240404=43, srv24660948=53, srv563594673=73, srv850074997=95, srv134833938=12, srv1960447602=37, srv82414453=93, srv227436634=48, srv538615922=69, srv1828513910=29, srv534944238=68, srv54508703=70, srv1829692473=31, srv456111782=65, srv1784902581=26, srv1458397652=15, srv461538518=66, srv37877662=60, srv1328114476=10, srv595272635=78, srv1346272732=11, srv380009089=61, srv165699305=21, srv405759472=63, srv1209517805=5, srv1682413768=23, srv1842556185=32, srv1201213592=4, srv1820937409=28, srv820742007=92, srv1466597012=16, srv819866202=91, srv1698542698=24, srv593438444=77, srv703125158=85, srv967592295=99, srv849973048=94, srv2054099354=42, srv584677683=75, srv240766402=52, srv23138020=49, srv1951490590=35, srv1420122369=14, srv1606002261=20, srv2089146179=45, srv502482839=67, srv1958146009=36, srv239831769=51, srv320201703=56, srv590608334=76, srv364135130=58, srv1101140186=1, srv1318828499=9, srv1874597875=33, srv2005386537=40, srv17659757=25, srv579192810=74, srv74181935=87, srv1302358587=7, srv615639059=79, srv898419999=96, srv2077003024=44, srv237913704=50, srv1240882252=6, srv1578969763=19, srv1196638697=3, srv932835779=98, srv223102748=47, srv1829394076=30, srv1663750=22, srv645862840=83, srv196255433=38, srv548866751=71, srv632749287=81, srv779999860=89, srv819333271=90, srv910293841=97, srv1049834700=0, srv247933306=54, srv1970290355=39, srv339598953=57, srv374152917=59, srv266409999=55, srv550557679=72, srv70239008=84, srv1414104133=13, srv627857943=80, srv150177733=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-26 01:58:46,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-26 01:58:46,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-26 01:58:46,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-26 01:58:46,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-26 01:58:46,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-26 01:58:46,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-26 01:58:46,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-26 01:58:46,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-26 01:58:46,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-26 01:58:46,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-26 01:58:46,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-26 01:58:46,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-26 01:58:46,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-26 01:58:46,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-26 01:58:46,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-26 01:58:46,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-26 01:58:46,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-26 01:58:46,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-26 01:58:46,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-26 01:58:46,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-26 01:58:46,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-26 01:58:46,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-26 01:58:46,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-26 01:58:46,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-26 01:58:46,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-26 01:58:46,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-26 01:58:46,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-26 01:58:46,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-26 01:58:46,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-26 01:58:46,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-26 01:58:46,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-26 01:58:46,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-26 01:58:46,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-26 01:58:46,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-26 01:58:46,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-26 01:58:46,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-26 01:58:46,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-26 01:58:46,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-26 01:58:46,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-26 01:58:46,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-26 01:58:46,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-26 01:58:46,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-26 01:58:46,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-26 01:58:46,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-26 01:58:46,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-26 01:58:46,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-26 01:58:46,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-26 01:58:46,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-26 01:58:46,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-26 01:58:46,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-26 01:58:46,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-26 01:58:46,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-26 01:58:46,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-26 01:58:46,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-26 01:58:46,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-26 01:58:46,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-26 01:58:46,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-26 01:58:46,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-26 01:58:46,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-26 01:58:46,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-26 01:58:46,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-26 01:58:46,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-26 01:58:46,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-26 01:58:46,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-26 01:58:46,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-26 01:58:46,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-26 01:58:46,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-26 01:58:46,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-26 01:58:46,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-26 01:58:46,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-26 01:58:46,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-26 01:58:46,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-26 01:58:46,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-26 01:58:46,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-26 01:58:46,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-26 01:58:46,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-26 01:58:46,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-26 01:58:46,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-26 01:58:46,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-26 01:58:46,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-26 01:58:46,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-26 01:58:46,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-26 01:58:46,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-26 01:58:46,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-26 01:58:46,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-26 01:58:46,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-26 01:58:46,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-26 01:58:46,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-26 01:58:46,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-26 01:58:46,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-26 01:58:46,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-26 01:58:46,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-26 01:58:46,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-26 01:58:46,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-26 01:58:46,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-26 01:58:46,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-26 01:58:46,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-26 01:58:46,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-26 01:58:46,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-26 01:58:46,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-26 01:58:46,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-26 01:58:46,617 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=9.085728464020627E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-26 01:58:46,617 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv237913704:82 , srv1101140186:83 , srv1958146009:83 , srv196255433:83 , srv2014494699:83 , srv24660948:83 , srv339598953:83 , srv579192810:83 , srv590608334:83 , srv74181935:83 , srv819866202:83 , srv82414453:83 , srv1049834700:84 , srv1201213592:84 , srv1240882252:84 , srv1318828499:84 , srv1414104133:84 , srv1420122369:84 , srv1458397652:84 , srv1578969763:84 , srv1606002261:84 , srv1698542698:84 , srv17659757:84 , srv1784902581:84 , srv178715911:84 , srv1820937409:84 , srv1828513910:84 , srv1829692473:84 , srv1842556185:84 , srv1951490590:84 , srv2089146179:84 , srv239831769:84 , srv320201703:84 , srv364135130:84 , srv37877662:84 , srv380009089:84 , srv456111782:84 , srv538615922:84 , srv54508703:84 , srv593438444:84 , srv595272635:84 , srv627857943:84 , srv632749287:84 , srv643927896:84 , srv70239008:84 , srv703125158:84 , srv73850172:84 , srv752767864:84 , srv849973048:84 , srv898419999:84 , srv910293841:84 , srv932835779:84 , srv1196638697:85 , srv1209517805:85 , srv1302358587:85 , srv1310157919:85 , srv1328114476:85 , srv1346272732:85 , srv134833938:85 , srv1466597012:85 , srv1477154116:85 , srv150177733:85 , srv165699305:85 , srv1663750:85 , srv1682413768:85 , srv1829394076:85 , srv1874597875:85 , srv1878564055:85 , srv1960447602:85 , srv1970290355:85 , srv2005386537:85 , srv2054099354:85 , srv2054240404:85 , srv2077003024:85 , srv2120395334:85 , srv223102748:85 , srv227436634:85 , srv23138020:85 , srv240766402:85 , srv247933306:85 , srv266409999:85 , srv374152917:85 , srv388969810:85 , srv405759472:85 , srv423123302:85 , srv461538518:85 , srv502482839:85 , srv550557679:85 , srv563594673:85 , srv584677683:85 , srv615639059:85 , srv645862840:85 , srv779999860:85 , srv819333271:85 , srv820742007:85 , srv850074997:85 , srv967592295:85 , srv1146929175:86 , srv548866751:86 , srv534944238:648 }
2023-09-26 01:58:46,639 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=188 (was 135) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 166), AvailableMemoryMB=7841 (was 8273)
2023-09-26 01:58:46,646 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=188, ProcessCount=166, AvailableMemoryMB=7841
2023-09-26 01:58:46,646 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-26 01:58:46,647 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-09-26 01:58:46,658 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1177608192:84 , srv1229880699:84 , srv1257595154:84 , srv1498777314:84 , srv155447985:84 , srv1619696264:84 , srv1801263616:84 , srv1819721543:84 , srv1846423758:84 , srv1919558917:84 , srv1944834521:84 , srv2023015003:84 , srv2030250357:84 , srv2047668783:84 , srv2129168100:84 , srv263163127:84 , srv265974198:84 , srv358646744:84 , srv400321042:84 , srv452649191:84 , srv541437006:84 , srv585421849:84 , srv605912620:84 , srv648718571:84 , srv713875144:84 , srv724042625:84 , srv796946154:84 , srv814594959:84 , srv906363650:84 , srv1906833013:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-09-26 01:58:46,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-26 01:58:46,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1229880699=1, srv713875144=25, srv1177608192=0, srv358646744=18, srv263163127=16, srv1257595154=2, srv2047668783=14, srv1498777314=3, srv155447985=4, srv1919558917=10, srv2023015003=12, srv814594959=28, srv796946154=27, srv2030250357=13, srv906363650=29, srv2129168100=15, srv1619696264=5, srv605912620=23, srv724042625=26, srv541437006=21, srv585421849=22, srv1846423758=8, srv452649191=20, srv1819721543=7, srv1906833013=9, srv648718571=24, srv1801263616=6, srv1944834521=11, srv265974198=17, srv400321042=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-26 01:58:46,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-26 01:58:46,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-26 01:58:46,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-26 01:58:46,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-26 01:58:46,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-26 01:58:46,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-26 01:58:46,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-26 01:58:46,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-26 01:58:46,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-26 01:58:46,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-26 01:58:46,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-26 01:58:46,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-26 01:58:46,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-26 01:58:46,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-26 01:58:46,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-26 01:58:46,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-26 01:58:46,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-26 01:58:46,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-26 01:58:46,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-26 01:58:46,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-26 01:58:46,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-26 01:58:46,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-26 01:58:46,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-26 01:58:46,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-26 01:58:46,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-26 01:58:46,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-26 01:58:46,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-26 01:58:46,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-26 01:58:46,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-26 01:58:46,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-26 01:58:46,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-09-26 01:58:46,701 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0589222799112583, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-26 01:58:46,702 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912378705444291, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0589222799112583, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-09-26 02:00:46,680 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 10023812 different iterations.  Found a solution that moves 1837 regions; Going from a computed imbalance of 0.9912378705444291 to a new imbalance of 4.2876496417733574E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6803703703703704, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-26 02:00:46,685 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1177608192:90 , srv1229880699:90 , srv1257595154:90 , srv1498777314:90 , srv155447985:90 , srv1619696264:90 , srv1801263616:90 , srv1819721543:90 , srv1846423758:90 , srv1906833013:90 , srv1919558917:90 , srv1944834521:90 , srv2023015003:90 , srv2030250357:90 , srv2047668783:90 , srv2129168100:90 , srv263163127:90 , srv265974198:90 , srv358646744:90 , srv400321042:90 , srv452649191:90 , srv541437006:90 , srv585421849:90 , srv605912620:90 , srv648718571:90 , srv713875144:90 , srv724042625:90 , srv796946154:90 , srv814594959:90 , srv906363650:90 }
2023-09-26 02:00:46,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-26 02:00:46,686 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1229880699=1, srv713875144=25, srv1177608192=0, srv358646744=18, srv263163127=16, srv1257595154=2, srv2047668783=14, srv1498777314=3, srv155447985=4, srv1919558917=10, srv2023015003=12, srv814594959=28, srv796946154=27, srv2030250357=13, srv906363650=29, srv2129168100=15, srv1619696264=5, srv605912620=23, srv724042625=26, srv541437006=21, srv585421849=22, srv1846423758=8, srv452649191=20, srv1819721543=7, srv1906833013=9, srv648718571=24, srv1801263616=6, srv1944834521=11, srv265974198=17, srv400321042=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-26 02:00:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-26 02:00:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-26 02:00:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-26 02:00:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-26 02:00:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-26 02:00:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-26 02:00:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-26 02:00:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-26 02:00:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-26 02:00:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-26 02:00:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-26 02:00:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-26 02:00:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-26 02:00:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-26 02:00:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-26 02:00:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-26 02:00:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-26 02:00:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-26 02:00:46,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-26 02:00:46,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-26 02:00:46,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-26 02:00:46,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-26 02:00:46,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-26 02:00:46,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-26 02:00:46,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-26 02:00:46,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-26 02:00:46,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-26 02:00:46,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-26 02:00:46,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-26 02:00:46,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-26 02:00:46,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-09-26 02:00:46,699 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-26 02:00:46,699 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1177608192:90 , srv1229880699:90 , srv1257595154:90 , srv1498777314:90 , srv155447985:90 , srv1619696264:90 , srv1801263616:90 , srv1819721543:90 , srv1846423758:90 , srv1906833013:90 , srv1919558917:90 , srv1944834521:90 , srv2023015003:90 , srv2030250357:90 , srv2047668783:90 , srv2129168100:90 , srv263163127:90 , srv265974198:90 , srv358646744:90 , srv400321042:90 , srv452649191:90 , srv541437006:90 , srv585421849:90 , srv605912620:90 , srv648718571:90 , srv713875144:90 , srv724042625:90 , srv796946154:90 , srv814594959:90 , srv906363650:90 }
2023-09-26 02:00:46,710 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=122 (was 188), ProcessCount=163 (was 166), AvailableMemoryMB=8720 (was 7841) - AvailableMemoryMB LEAK? -