2024-03-15 18:53:25,879 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-03-15 18:53:26,164 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-03-15 18:53:26,268 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-03-15 18:53:26,328 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-15 18:53:26,335 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-15 18:53:26,350 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=225, ProcessCount=170, AvailableMemoryMB=4668
2024-03-15 18:53:26,352 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-15 18:53:26,352 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-15 18:53:26,478 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1042963007:84 , srv1097848214:84 , srv1185962669:84 , srv1198341721:84 , srv1199383193:84 , srv1238300576:84 , srv1251066494:84 , srv1273238834:84 , srv1286297603:84 , srv1322619401:84 , srv1379367633:84 , srv1380898481:84 , srv1434800321:84 , srv1439966720:84 , srv1450166210:84 , srv1451476386:84 , srv1463124868:84 , srv1473840153:84 , srv1499284603:84 , srv1504306255:84 , srv1509430142:84 , srv153146935:84 , srv1581364861:84 , srv1594606193:84 , srv1595824932:84 , srv1623632017:84 , srv1623654077:84 , srv1630690221:84 , srv1635286883:84 , srv1659990052:84 , srv1664425027:84 , srv1669131229:84 , srv1715097040:84 , srv1722847927:84 , srv1756271896:84 , srv1769472022:84 , srv1776039349:84 , srv1871522006:84 , srv1891518889:84 , srv1944536138:84 , srv1965204166:84 , srv199346244:84 , srv201361064:84 , srv2025734038:84 , srv2039364009:84 , srv2069401623:84 , srv2070103440:84 , srv209714856:84 , srv2138811599:84 , srv2143585644:84 , srv291246251:84 , srv299344076:84 , srv306430723:84 , srv315407227:84 , srv34377132:84 , srv365579606:84 , srv381924643:84 , srv385764542:84 , srv38603165:84 , srv391085017:84 , srv407341666:84 , srv414887426:84 , srv467883906:84 , srv508685881:84 , srv512721246:84 , srv515549340:84 , srv525756889:84 , srv526623844:84 , srv527175959:84 , srv530240784:84 , srv553745420:84 , srv557957781:84 , srv571516055:84 , srv577504057:84 , srv580819471:84 , srv59443007:84 , srv617158084:84 , srv62731174:84 , srv628792546:84 , srv670751110:84 , srv6925371:84 , srv699268918:84 , srv713608329:84 , srv746759393:84 , srv7671403:84 , srv767582430:84 , srv769007821:84 , srv773798363:84 , srv78550135:84 , srv798037448:84 , srv840672661:84 , srv843980445:84 , srv885484897:84 , srv920156928:84 , srv922823792:84 , srv965134922:84 , srv966279153:84 , srv97166646:84 , srv974746182:84 , srv254818335:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-03-15 18:53:26,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-15 18:53:26,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv670751110=80, srv2039364009=44, srv843980445=92, srv254818335=50, srv1669131229=31, srv1715097040=32, srv467883906=63, srv1463124868=16, srv1273238834=7, srv769007821=87, srv1769472022=35, srv798037448=90, srv974746182=99, srv965134922=96, srv1635286883=28, srv1251066494=6, srv557957781=72, srv1434800321=12, srv97166646=98, srv1623632017=25, srv7671403=85, srv553745420=71, srv315407227=54, srv1756271896=34, srv299344076=52, srv1439966720=13, srv512721246=65, srv391085017=60, srv2069401623=45, srv508685881=64, srv1042963007=0, srv1509430142=20, srv1944536138=39, srv773798363=88, srv201361064=42, srv966279153=97, srv1594606193=23, srv407341666=61, srv525756889=67, srv1199383193=4, srv34377132=55, srv1776039349=36, srv840672661=91, srv1664425027=30, srv6925371=81, srv1659990052=29, srv1473840153=17, srv1380898481=11, srv385764542=58, srv1286297603=8, srv291246251=51, srv1451476386=15, srv628792546=79, srv577504057=74, srv1623654077=26, srv209714856=47, srv515549340=66, srv306430723=53, srv767582430=86, srv199346244=41, srv1581364861=22, srv59443007=76, srv2143585644=49, srv530240784=70, srv571516055=73, srv1238300576=5, srv580819471=75, srv1871522006=37, srv1630690221=27, srv2138811599=48, srv2070103440=46, srv62731174=78, srv2025734038=43, srv1504306255=19, srv1595824932=24, srv1322619401=9, srv1891518889=38, srv527175959=69, srv885484897=93, srv1379367633=10, srv920156928=94, srv922823792=95, srv699268918=82, srv746759393=84, srv1722847927=33, srv414887426=62, srv1450166210=14, srv38603165=59, srv1097848214=1, srv153146935=21, srv526623844=68, srv713608329=83, srv78550135=89, srv1185962669=2, srv1198341721=3, srv1499284603=18, srv617158084=77, srv365579606=56, srv381924643=57, srv1965204166=40} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-15 18:53:26,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-15 18:53:26,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-15 18:53:26,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-15 18:53:26,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-15 18:53:26,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-15 18:53:26,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-15 18:53:26,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-15 18:53:26,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-15 18:53:26,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-15 18:53:26,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-15 18:53:26,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-15 18:53:26,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-15 18:53:26,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-15 18:53:26,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-15 18:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-15 18:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-15 18:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-15 18:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-15 18:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-15 18:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-15 18:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-15 18:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-15 18:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-15 18:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-15 18:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-15 18:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-15 18:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-15 18:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-15 18:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-15 18:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-15 18:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-15 18:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-15 18:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-15 18:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-15 18:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-15 18:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-15 18:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-15 18:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-15 18:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-15 18:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-15 18:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-15 18:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-15 18:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-15 18:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-15 18:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-15 18:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-15 18:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-15 18:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-15 18:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-15 18:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-15 18:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-15 18:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-15 18:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-15 18:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-15 18:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-15 18:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-15 18:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-15 18:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-15 18:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-15 18:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-15 18:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-15 18:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-15 18:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-15 18:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-15 18:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-15 18:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-15 18:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-15 18:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-15 18:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-15 18:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-15 18:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-15 18:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-15 18:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-15 18:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-15 18:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-15 18:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-15 18:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-15 18:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-15 18:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-15 18:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-15 18:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-15 18:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-15 18:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-15 18:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-15 18:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-15 18:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-15 18:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-15 18:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-15 18:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-15 18:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-15 18:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-15 18:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-15 18:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-15 18:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-15 18:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-15 18:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-15 18:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-15 18:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-15 18:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-15 18:53:26,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-15 18:53:26,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-15 18:53:26,763 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-15 18:53:26,764 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-15 18:53:26,764 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-15 18:53:27,129 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 531 ms to try 5000 different iterations.  Found a solution that moves 4678 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1849908343709759. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03996506767470401, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06141430754895126, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5197777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03996506767470401, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18033333333333335, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24591666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-15 18:53:27,144 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1473840153:80 , srv530240784:80 , srv2138811599:81 , srv525756889:81 , srv974746182:81 , srv1199383193:82 , srv1451476386:82 , srv526623844:82 , srv1595824932:83 , srv1715097040:83 , srv407341666:83 , srv1434800321:84 , srv1439966720:84 , srv1463124868:84 , srv1756271896:84 , srv1776039349:84 , srv201361064:84 , srv299344076:84 , srv467883906:84 , srv713608329:84 , srv920156928:84 , srv922823792:84 , srv966279153:84 , srv1198341721:85 , srv1251066494:85 , srv1286297603:85 , srv1499284603:85 , srv153146935:85 , srv1630690221:85 , srv1664425027:85 , srv1965204166:85 , srv385764542:85 , srv62731174:85 , srv670751110:85 , srv769007821:85 , srv773798363:85 , srv965134922:85 , srv1273238834:86 , srv1322619401:86 , srv1380898481:86 , srv1504306255:86 , srv1623654077:86 , srv1669131229:86 , srv1722847927:86 , srv1769472022:86 , srv2039364009:86 , srv306430723:86 , srv315407227:86 , srv365579606:86 , srv414887426:86 , srv508685881:86 , srv59443007:86 , srv617158084:86 , srv699268918:86 , srv1238300576:87 , srv1594606193:87 , srv199346244:87 , srv209714856:87 , srv34377132:87 , srv391085017:87 , srv527175959:87 , srv557957781:87 , srv628792546:87 , srv746759393:87 , srv7671403:87 , srv798037448:87 , srv1379367633:88 , srv1509430142:88 , srv1623632017:88 , srv1635286883:88 , srv1659990052:88 , srv1944536138:88 , srv2025734038:88 , srv2069401623:88 , srv515549340:88 , srv571516055:88 , srv840672661:88 , srv843980445:88 , srv885484897:88 , srv1450166210:89 , srv1891518889:89 , srv2070103440:89 , srv2143585644:89 , srv381924643:89 , srv553745420:89 , srv580819471:89 , srv1097848214:90 , srv1581364861:90 , srv1871522006:90 , srv291246251:90 , srv512721246:90 , srv577504057:90 , srv6925371:90 , srv78550135:91 , srv1042963007:92 , srv38603165:92 , srv1185962669:93 , srv97166646:93 , srv767582430:95 , srv254818335:445 }
2024-03-15 18:53:27,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-15 18:53:27,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv670751110=80, srv2039364009=44, srv843980445=92, srv254818335=50, srv1669131229=31, srv1715097040=32, srv467883906=63, srv1463124868=16, srv1273238834=7, srv769007821=87, srv1769472022=35, srv798037448=90, srv974746182=99, srv965134922=96, srv1635286883=28, srv1251066494=6, srv557957781=72, srv1434800321=12, srv97166646=98, srv1623632017=25, srv7671403=85, srv553745420=71, srv315407227=54, srv1756271896=34, srv299344076=52, srv1439966720=13, srv512721246=65, srv391085017=60, srv2069401623=45, srv508685881=64, srv1042963007=0, srv1509430142=20, srv1944536138=39, srv773798363=88, srv201361064=42, srv966279153=97, srv1594606193=23, srv407341666=61, srv525756889=67, srv1199383193=4, srv34377132=55, srv1776039349=36, srv840672661=91, srv1664425027=30, srv6925371=81, srv1659990052=29, srv1473840153=17, srv1380898481=11, srv385764542=58, srv1286297603=8, srv291246251=51, srv1451476386=15, srv628792546=79, srv577504057=74, srv1623654077=26, srv209714856=47, srv515549340=66, srv306430723=53, srv767582430=86, srv199346244=41, srv1581364861=22, srv59443007=76, srv2143585644=49, srv530240784=70, srv571516055=73, srv1238300576=5, srv580819471=75, srv1871522006=37, srv1630690221=27, srv2138811599=48, srv2070103440=46, srv62731174=78, srv2025734038=43, srv1504306255=19, srv1595824932=24, srv1322619401=9, srv1891518889=38, srv527175959=69, srv885484897=93, srv1379367633=10, srv920156928=94, srv922823792=95, srv699268918=82, srv746759393=84, srv1722847927=33, srv414887426=62, srv1450166210=14, srv38603165=59, srv1097848214=1, srv153146935=21, srv526623844=68, srv713608329=83, srv78550135=89, srv1185962669=2, srv1198341721=3, srv1499284603=18, srv617158084=77, srv365579606=56, srv381924643=57, srv1965204166=40} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-15 18:53:27,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-15 18:53:27,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-15 18:53:27,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-15 18:53:27,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-15 18:53:27,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-15 18:53:27,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-15 18:53:27,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-15 18:53:27,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-15 18:53:27,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-15 18:53:27,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-15 18:53:27,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-15 18:53:27,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-15 18:53:27,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-15 18:53:27,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-15 18:53:27,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-15 18:53:27,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-15 18:53:27,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-15 18:53:27,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-15 18:53:27,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-15 18:53:27,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-15 18:53:27,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-15 18:53:27,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-15 18:53:27,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-15 18:53:27,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-15 18:53:27,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-15 18:53:27,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-15 18:53:27,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-15 18:53:27,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-15 18:53:27,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-15 18:53:27,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-15 18:53:27,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-15 18:53:27,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-15 18:53:27,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-15 18:53:27,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-15 18:53:27,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-15 18:53:27,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-15 18:53:27,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-15 18:53:27,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-15 18:53:27,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-15 18:53:27,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-15 18:53:27,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-15 18:53:27,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-15 18:53:27,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-15 18:53:27,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-15 18:53:27,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-15 18:53:27,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-15 18:53:27,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-15 18:53:27,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-15 18:53:27,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-15 18:53:27,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-15 18:53:27,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-15 18:53:27,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-15 18:53:27,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-15 18:53:27,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-15 18:53:27,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-15 18:53:27,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-15 18:53:27,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-15 18:53:27,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-15 18:53:27,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-15 18:53:27,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-15 18:53:27,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-15 18:53:27,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-15 18:53:27,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-15 18:53:27,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-15 18:53:27,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-15 18:53:27,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-15 18:53:27,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-15 18:53:27,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-15 18:53:27,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-15 18:53:27,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-15 18:53:27,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-15 18:53:27,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-15 18:53:27,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-15 18:53:27,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-15 18:53:27,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-15 18:53:27,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-15 18:53:27,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-15 18:53:27,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-15 18:53:27,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-15 18:53:27,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-15 18:53:27,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-15 18:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-15 18:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-15 18:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-15 18:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-15 18:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-15 18:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-15 18:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-15 18:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-15 18:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-15 18:53:27,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-15 18:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-15 18:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-15 18:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-15 18:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-15 18:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-15 18:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-15 18:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-15 18:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-15 18:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-15 18:53:27,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-15 18:53:27,359 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03996506767470401, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06141430754895126, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03996506767470401, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18033333333333335, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24591666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-15 18:53:27,360 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-15 18:53:27,360 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1849580783148667, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03996506767470401, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06141430754895126, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03996506767470401, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18033333333333335, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24591666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-15 18:53:27,619 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 408 ms to try 5000 different iterations.  Found a solution that moves 2691 regions; Going from a computed imbalance of 0.1849580783148667 to a new imbalance of 0.016988704537442223. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02160708663360516); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03513641844631533, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.299, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02160708663360516); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04975, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-15 18:53:27,627 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv767582430:77 , srv1185962669:81 , srv1594606193:81 , srv1891518889:81 , srv38603165:81 , srv553745420:81 , srv577504057:81 , srv97166646:81 , srv1042963007:82 , srv1097848214:82 , srv1379367633:82 , srv1581364861:82 , srv2070103440:82 , srv512721246:82 , srv515549340:82 , srv580819471:82 , srv1238300576:83 , srv1450166210:83 , srv1871522006:83 , srv1944536138:83 , srv381924643:83 , srv571516055:83 , srv6925371:83 , srv840672661:83 , srv1504306255:84 , srv1635286883:84 , srv1659990052:84 , srv2025734038:84 , srv2069401623:84 , srv209714856:84 , srv2143585644:84 , srv291246251:84 , srv34377132:84 , srv7671403:84 , srv1273238834:85 , srv1509430142:85 , srv1623654077:85 , srv1630690221:85 , srv1769472022:85 , srv199346244:85 , srv628792546:85 , srv713608329:85 , srv78550135:85 , srv798037448:85 , srv885484897:85 , srv1623632017:86 , srv1715097040:86 , srv1776039349:86 , srv2039364009:86 , srv306430723:86 , srv391085017:86 , srv508685881:86 , srv557957781:86 , srv59443007:86 , srv746759393:86 , srv769007821:86 , srv843980445:86 , srv1198341721:87 , srv1251066494:87 , srv1322619401:87 , srv1451476386:87 , srv1463124868:87 , srv153146935:87 , srv1669131229:87 , srv1722847927:87 , srv1965204166:87 , srv315407227:87 , srv365579606:87 , srv385764542:87 , srv414887426:87 , srv527175959:87 , srv617158084:87 , srv62731174:87 , srv670751110:87 , srv773798363:87 , srv965134922:87 , srv966279153:87 , srv1595824932:88 , srv1664425027:88 , srv1756271896:88 , srv299344076:88 , srv467883906:88 , srv1199383193:89 , srv1380898481:89 , srv1439966720:89 , srv1499284603:89 , srv201361064:89 , srv526623844:89 , srv699268918:89 , srv920156928:89 , srv922823792:89 , srv974746182:89 , srv1286297603:90 , srv1434800321:90 , srv407341666:90 , srv2138811599:91 , srv525756889:91 , srv530240784:92 , srv1473840153:94 , srv254818335:521 }
2024-03-15 18:53:27,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-15 18:53:27,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv670751110=80, srv2039364009=44, srv843980445=92, srv254818335=50, srv1669131229=31, srv1715097040=32, srv467883906=63, srv1463124868=16, srv1273238834=7, srv769007821=87, srv1769472022=35, srv798037448=90, srv974746182=99, srv965134922=96, srv1635286883=28, srv1251066494=6, srv557957781=72, srv1434800321=12, srv97166646=98, srv1623632017=25, srv7671403=85, srv553745420=71, srv315407227=54, srv1756271896=34, srv299344076=52, srv1439966720=13, srv512721246=65, srv391085017=60, srv2069401623=45, srv508685881=64, srv1042963007=0, srv1509430142=20, srv1944536138=39, srv773798363=88, srv201361064=42, srv966279153=97, srv1594606193=23, srv407341666=61, srv525756889=67, srv1199383193=4, srv34377132=55, srv1776039349=36, srv840672661=91, srv1664425027=30, srv6925371=81, srv1659990052=29, srv1473840153=17, srv1380898481=11, srv385764542=58, srv1286297603=8, srv291246251=51, srv1451476386=15, srv628792546=79, srv577504057=74, srv1623654077=26, srv209714856=47, srv515549340=66, srv306430723=53, srv767582430=86, srv199346244=41, srv1581364861=22, srv59443007=76, srv2143585644=49, srv530240784=70, srv571516055=73, srv1238300576=5, srv580819471=75, srv1871522006=37, srv1630690221=27, srv2138811599=48, srv2070103440=46, srv62731174=78, srv2025734038=43, srv1504306255=19, srv1595824932=24, srv1322619401=9, srv1891518889=38, srv527175959=69, srv885484897=93, srv1379367633=10, srv920156928=94, srv922823792=95, srv699268918=82, srv746759393=84, srv1722847927=33, srv414887426=62, srv1450166210=14, srv38603165=59, srv1097848214=1, srv153146935=21, srv526623844=68, srv713608329=83, srv78550135=89, srv1185962669=2, srv1198341721=3, srv1499284603=18, srv617158084=77, srv365579606=56, srv381924643=57, srv1965204166=40} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-15 18:53:27,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-15 18:53:27,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-15 18:53:27,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-15 18:53:27,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-15 18:53:27,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-15 18:53:27,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-15 18:53:27,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-15 18:53:27,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-15 18:53:27,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-15 18:53:27,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-15 18:53:27,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-15 18:53:27,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-15 18:53:27,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-15 18:53:27,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-15 18:53:27,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-15 18:53:27,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-15 18:53:27,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-15 18:53:27,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-15 18:53:27,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-15 18:53:27,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-15 18:53:27,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-15 18:53:27,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-15 18:53:27,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-15 18:53:27,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-15 18:53:27,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-15 18:53:27,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-15 18:53:27,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-15 18:53:27,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-15 18:53:27,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-15 18:53:27,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-15 18:53:27,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-15 18:53:27,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-15 18:53:27,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-15 18:53:27,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-15 18:53:27,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-15 18:53:27,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-15 18:53:27,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-15 18:53:27,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-15 18:53:27,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-15 18:53:27,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-15 18:53:27,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-15 18:53:27,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-15 18:53:27,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-15 18:53:27,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-15 18:53:27,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-15 18:53:27,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-15 18:53:27,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-15 18:53:27,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-15 18:53:27,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-15 18:53:27,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-15 18:53:27,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-15 18:53:27,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-15 18:53:27,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-15 18:53:27,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-15 18:53:27,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-15 18:53:27,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-15 18:53:27,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-15 18:53:27,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-15 18:53:27,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-15 18:53:27,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-15 18:53:27,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-15 18:53:27,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-15 18:53:27,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-15 18:53:27,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-15 18:53:27,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-15 18:53:27,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-15 18:53:27,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-15 18:53:27,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-15 18:53:27,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-15 18:53:27,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-15 18:53:27,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-15 18:53:27,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-15 18:53:27,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-15 18:53:27,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-15 18:53:27,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-15 18:53:27,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-15 18:53:27,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-15 18:53:27,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-15 18:53:27,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-15 18:53:27,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-15 18:53:27,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-15 18:53:27,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-15 18:53:27,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-15 18:53:27,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-15 18:53:27,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-15 18:53:27,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-15 18:53:27,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-15 18:53:27,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-15 18:53:27,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-15 18:53:27,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-15 18:53:27,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-15 18:53:27,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-15 18:53:27,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-15 18:53:27,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-15 18:53:27,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-15 18:53:27,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-15 18:53:27,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-15 18:53:27,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-15 18:53:27,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-15 18:53:27,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-15 18:53:27,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-15 18:53:27,811 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02160708663360516); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03513641844631533, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02160708663360516); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04975, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-15 18:53:27,811 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-15 18:53:27,812 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01696986175270731, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02160708663360516); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03513641844631533, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02160708663360516); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04975, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-15 18:53:28,051 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 373 ms to try 5000 different iterations.  Found a solution that moves 1058 regions; Going from a computed imbalance of 0.01696986175270731 to a new imbalance of 8.520655776942989E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007786591126529124); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01097901273060199); MoveCostFunction : (multiplier=7.0, imbalance=0.11755555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007786591126529124); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-15 18:53:28,056 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1286297603:82 , srv1380898481:82 , srv78550135:82 , srv1042963007:83 , srv1185962669:83 , srv1322619401:83 , srv1473840153:83 , srv1499284603:83 , srv291246251:83 , srv527175959:83 , srv699268918:83 , srv1251066494:84 , srv1434800321:84 , srv1509430142:84 , srv1664425027:84 , srv1871522006:84 , srv2143585644:84 , srv306430723:84 , srv315407227:84 , srv391085017:84 , srv407341666:84 , srv557957781:84 , srv59443007:84 , srv767582430:84 , srv843980445:84 , srv885484897:84 , srv922823792:84 , srv1097848214:85 , srv1198341721:85 , srv1439966720:85 , srv1450166210:85 , srv1463124868:85 , srv153146935:85 , srv1635286883:85 , srv1669131229:85 , srv1756271896:85 , srv1769472022:85 , srv1965204166:85 , srv199346244:85 , srv2039364009:85 , srv299344076:85 , srv365579606:85 , srv381924643:85 , srv385764542:85 , srv38603165:85 , srv414887426:85 , srv467883906:85 , srv512721246:85 , srv525756889:85 , srv571516055:85 , srv580819471:85 , srv617158084:85 , srv62731174:85 , srv628792546:85 , srv6925371:85 , srv746759393:85 , srv798037448:85 , srv965134922:85 , srv97166646:85 , srv1199383193:86 , srv1273238834:86 , srv1581364861:86 , srv1623632017:86 , srv1623654077:86 , srv1659990052:86 , srv1722847927:86 , srv201361064:86 , srv2025734038:86 , srv2069401623:86 , srv209714856:86 , srv2138811599:86 , srv34377132:86 , srv508685881:86 , srv526623844:86 , srv530240784:86 , srv577504057:86 , srv670751110:86 , srv769007821:86 , srv773798363:86 , srv840672661:86 , srv920156928:86 , srv966279153:86 , srv1238300576:87 , srv1379367633:87 , srv1504306255:87 , srv1595824932:87 , srv1891518889:87 , srv1944536138:87 , srv2070103440:87 , srv515549340:87 , srv7671403:87 , srv974746182:87 , srv1451476386:88 , srv1630690221:88 , srv1715097040:88 , srv1776039349:88 , srv553745420:88 , srv713608329:88 , srv1594606193:89 , srv254818335:561 }
2024-03-15 18:53:28,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-15 18:53:28,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv670751110=80, srv2039364009=44, srv843980445=92, srv254818335=50, srv1669131229=31, srv1715097040=32, srv467883906=63, srv1463124868=16, srv1273238834=7, srv769007821=87, srv1769472022=35, srv798037448=90, srv974746182=99, srv965134922=96, srv1635286883=28, srv1251066494=6, srv557957781=72, srv1434800321=12, srv97166646=98, srv1623632017=25, srv7671403=85, srv553745420=71, srv315407227=54, srv1756271896=34, srv299344076=52, srv1439966720=13, srv512721246=65, srv391085017=60, srv2069401623=45, srv508685881=64, srv1042963007=0, srv1509430142=20, srv1944536138=39, srv773798363=88, srv201361064=42, srv966279153=97, srv1594606193=23, srv407341666=61, srv525756889=67, srv1199383193=4, srv34377132=55, srv1776039349=36, srv840672661=91, srv1664425027=30, srv6925371=81, srv1659990052=29, srv1473840153=17, srv1380898481=11, srv385764542=58, srv1286297603=8, srv291246251=51, srv1451476386=15, srv628792546=79, srv577504057=74, srv1623654077=26, srv209714856=47, srv515549340=66, srv306430723=53, srv767582430=86, srv199346244=41, srv1581364861=22, srv59443007=76, srv2143585644=49, srv530240784=70, srv571516055=73, srv1238300576=5, srv580819471=75, srv1871522006=37, srv1630690221=27, srv2138811599=48, srv2070103440=46, srv62731174=78, srv2025734038=43, srv1504306255=19, srv1595824932=24, srv1322619401=9, srv1891518889=38, srv527175959=69, srv885484897=93, srv1379367633=10, srv920156928=94, srv922823792=95, srv699268918=82, srv746759393=84, srv1722847927=33, srv414887426=62, srv1450166210=14, srv38603165=59, srv1097848214=1, srv153146935=21, srv526623844=68, srv713608329=83, srv78550135=89, srv1185962669=2, srv1198341721=3, srv1499284603=18, srv617158084=77, srv365579606=56, srv381924643=57, srv1965204166=40} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-15 18:53:28,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-15 18:53:28,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-15 18:53:28,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-15 18:53:28,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-15 18:53:28,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-15 18:53:28,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-15 18:53:28,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-15 18:53:28,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-15 18:53:28,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-15 18:53:28,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-15 18:53:28,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-15 18:53:28,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-15 18:53:28,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-15 18:53:28,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-15 18:53:28,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-15 18:53:28,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-15 18:53:28,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-15 18:53:28,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-15 18:53:28,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-15 18:53:28,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-15 18:53:28,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-15 18:53:28,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-15 18:53:28,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-15 18:53:28,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-15 18:53:28,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-15 18:53:28,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-15 18:53:28,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-15 18:53:28,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-15 18:53:28,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-15 18:53:28,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-15 18:53:28,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-15 18:53:28,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-15 18:53:28,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-15 18:53:28,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-15 18:53:28,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-15 18:53:28,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-15 18:53:28,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-15 18:53:28,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-15 18:53:28,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-15 18:53:28,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-15 18:53:28,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-15 18:53:28,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-15 18:53:28,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-15 18:53:28,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-15 18:53:28,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-15 18:53:28,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-15 18:53:28,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-15 18:53:28,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-15 18:53:28,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-15 18:53:28,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-15 18:53:28,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-15 18:53:28,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-15 18:53:28,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-15 18:53:28,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-15 18:53:28,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-15 18:53:28,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-15 18:53:28,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-15 18:53:28,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-15 18:53:28,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-15 18:53:28,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-15 18:53:28,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-15 18:53:28,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-15 18:53:28,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-15 18:53:28,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-15 18:53:28,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-15 18:53:28,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-15 18:53:28,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-15 18:53:28,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-15 18:53:28,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-15 18:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-15 18:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-15 18:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-15 18:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-15 18:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-15 18:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-15 18:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-15 18:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-15 18:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-15 18:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-15 18:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-15 18:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-15 18:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-15 18:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-15 18:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-15 18:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-15 18:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-15 18:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-15 18:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-15 18:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-15 18:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-15 18:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-15 18:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-15 18:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-15 18:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-15 18:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-15 18:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-15 18:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-15 18:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-15 18:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-15 18:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-15 18:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-15 18:53:28,191 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007786591126529124); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01097901273060199); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007786591126529124); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-15 18:53:28,191 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-15 18:53:28,191 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.446573033540764E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007786591126529124); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01097901273060199); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007786591126529124); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-15 18:53:28,529 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 439 ms to try 5000 different iterations.  Found a solution that moves 243 regions; Going from a computed imbalance of 8.446573033540764E-4 to a new imbalance of 3.3433112702119735E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.027, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-15 18:53:28,530 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1623632017:82 , srv201361064:82 , srv97166646:82 , srv1722847927:83 , srv2070103440:83 , srv920156928:83 , srv1434800321:84 , srv1439966720:84 , srv1581364861:84 , srv1595824932:84 , srv1623654077:84 , srv1630690221:84 , srv1659990052:84 , srv1664425027:84 , srv1669131229:84 , srv1776039349:84 , srv1944536138:84 , srv2025734038:84 , srv2069401623:84 , srv2138811599:84 , srv315407227:84 , srv365579606:84 , srv385764542:84 , srv414887426:84 , srv530240784:84 , srv553745420:84 , srv617158084:84 , srv670751110:84 , srv6925371:84 , srv699268918:84 , srv746759393:84 , srv7671403:84 , srv773798363:84 , srv78550135:84 , srv843980445:84 , srv1042963007:85 , srv1097848214:85 , srv1185962669:85 , srv1198341721:85 , srv1199383193:85 , srv1238300576:85 , srv1251066494:85 , srv1286297603:85 , srv1379367633:85 , srv1380898481:85 , srv1450166210:85 , srv1451476386:85 , srv1473840153:85 , srv1499284603:85 , srv1504306255:85 , srv153146935:85 , srv1635286883:85 , srv1715097040:85 , srv1756271896:85 , srv1871522006:85 , srv1891518889:85 , srv1965204166:85 , srv2039364009:85 , srv209714856:85 , srv2143585644:85 , srv291246251:85 , srv299344076:85 , srv34377132:85 , srv381924643:85 , srv38603165:85 , srv391085017:85 , srv407341666:85 , srv467883906:85 , srv508685881:85 , srv512721246:85 , srv515549340:85 , srv525756889:85 , srv526623844:85 , srv527175959:85 , srv557957781:85 , srv577504057:85 , srv62731174:85 , srv628792546:85 , srv713608329:85 , srv769007821:85 , srv840672661:85 , srv885484897:85 , srv922823792:85 , srv965134922:85 , srv966279153:85 , srv974746182:85 , srv1273238834:86 , srv1322619401:86 , srv1463124868:86 , srv1509430142:86 , srv1594606193:86 , srv1769472022:86 , srv199346244:86 , srv306430723:86 , srv571516055:86 , srv580819471:86 , srv767582430:86 , srv798037448:86 , srv59443007:87 , srv254818335:615 }
2024-03-15 18:53:28,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-15 18:53:28,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv670751110=80, srv2039364009=44, srv843980445=92, srv254818335=50, srv1669131229=31, srv1715097040=32, srv467883906=63, srv1463124868=16, srv1273238834=7, srv769007821=87, srv1769472022=35, srv798037448=90, srv974746182=99, srv965134922=96, srv1635286883=28, srv1251066494=6, srv557957781=72, srv1434800321=12, srv97166646=98, srv1623632017=25, srv7671403=85, srv553745420=71, srv315407227=54, srv1756271896=34, srv299344076=52, srv1439966720=13, srv512721246=65, srv391085017=60, srv2069401623=45, srv508685881=64, srv1042963007=0, srv1509430142=20, srv1944536138=39, srv773798363=88, srv201361064=42, srv966279153=97, srv1594606193=23, srv407341666=61, srv525756889=67, srv1199383193=4, srv34377132=55, srv1776039349=36, srv840672661=91, srv1664425027=30, srv6925371=81, srv1659990052=29, srv1473840153=17, srv1380898481=11, srv385764542=58, srv1286297603=8, srv291246251=51, srv1451476386=15, srv628792546=79, srv577504057=74, srv1623654077=26, srv209714856=47, srv515549340=66, srv306430723=53, srv767582430=86, srv199346244=41, srv1581364861=22, srv59443007=76, srv2143585644=49, srv530240784=70, srv571516055=73, srv1238300576=5, srv580819471=75, srv1871522006=37, srv1630690221=27, srv2138811599=48, srv2070103440=46, srv62731174=78, srv2025734038=43, srv1504306255=19, srv1595824932=24, srv1322619401=9, srv1891518889=38, srv527175959=69, srv885484897=93, srv1379367633=10, srv920156928=94, srv922823792=95, srv699268918=82, srv746759393=84, srv1722847927=33, srv414887426=62, srv1450166210=14, srv38603165=59, srv1097848214=1, srv153146935=21, srv526623844=68, srv713608329=83, srv78550135=89, srv1185962669=2, srv1198341721=3, srv1499284603=18, srv617158084=77, srv365579606=56, srv381924643=57, srv1965204166=40} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-15 18:53:28,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-15 18:53:28,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-15 18:53:28,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-15 18:53:28,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-15 18:53:28,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-15 18:53:28,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-15 18:53:28,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-15 18:53:28,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-15 18:53:28,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-15 18:53:28,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-15 18:53:28,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-15 18:53:28,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-15 18:53:28,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-15 18:53:28,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-15 18:53:28,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-15 18:53:28,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-15 18:53:28,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-15 18:53:28,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-15 18:53:28,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-15 18:53:28,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-15 18:53:28,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-15 18:53:28,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-15 18:53:28,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-15 18:53:28,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-15 18:53:28,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-15 18:53:28,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-15 18:53:28,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-15 18:53:28,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-15 18:53:28,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-15 18:53:28,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-15 18:53:28,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-15 18:53:28,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-15 18:53:28,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-15 18:53:28,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-15 18:53:28,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-15 18:53:28,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-15 18:53:28,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-15 18:53:28,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-15 18:53:28,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-15 18:53:28,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-15 18:53:28,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-15 18:53:28,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-15 18:53:28,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-15 18:53:28,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-15 18:53:28,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-15 18:53:28,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-15 18:53:28,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-15 18:53:28,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-15 18:53:28,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-15 18:53:28,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-15 18:53:28,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-15 18:53:28,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-15 18:53:28,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-15 18:53:28,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-15 18:53:28,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-15 18:53:28,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-15 18:53:28,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-15 18:53:28,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-15 18:53:28,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-15 18:53:28,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-15 18:53:28,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-15 18:53:28,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-15 18:53:28,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-15 18:53:28,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-15 18:53:28,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-15 18:53:28,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-15 18:53:28,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-15 18:53:28,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-15 18:53:28,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-15 18:53:28,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-15 18:53:28,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-15 18:53:28,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-15 18:53:28,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-15 18:53:28,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-15 18:53:28,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-15 18:53:28,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-15 18:53:28,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-15 18:53:28,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-15 18:53:28,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-15 18:53:28,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-15 18:53:28,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-15 18:53:28,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-15 18:53:28,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-15 18:53:28,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-15 18:53:28,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-15 18:53:28,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-15 18:53:28,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-15 18:53:28,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-15 18:53:28,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-15 18:53:28,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-15 18:53:28,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-15 18:53:28,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-15 18:53:28,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-15 18:53:28,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-15 18:53:28,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-15 18:53:28,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-15 18:53:28,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-15 18:53:28,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-15 18:53:28,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-15 18:53:28,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-15 18:53:28,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-15 18:53:28,661 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-15 18:53:28,661 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-15 18:53:28,661 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.1731590334752956E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-15 18:53:28,889 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 328 ms to try 5000 different iterations.  Found a solution that moves 24 regions; Going from a computed imbalance of 3.1731590334752956E-5 to a new imbalance of 5.391985018071482E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0011605177063713189); MoveCostFunction : (multiplier=7.0, imbalance=0.0026666666666666666); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-15 18:53:28,889 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1273238834:83 , srv1509430142:83 , srv1594606193:83 , srv199346244:83 , srv38603165:83 , srv508685881:83 , srv59443007:83 , srv798037448:83 , srv1042963007:84 , srv1097848214:84 , srv1185962669:84 , srv1198341721:84 , srv1199383193:84 , srv1238300576:84 , srv1286297603:84 , srv1322619401:84 , srv1379367633:84 , srv1380898481:84 , srv1434800321:84 , srv1439966720:84 , srv1450166210:84 , srv1451476386:84 , srv1463124868:84 , srv1473840153:84 , srv1499284603:84 , srv1504306255:84 , srv153146935:84 , srv1581364861:84 , srv1595824932:84 , srv1623632017:84 , srv1630690221:84 , srv1635286883:84 , srv1659990052:84 , srv1669131229:84 , srv1715097040:84 , srv1722847927:84 , srv1756271896:84 , srv1769472022:84 , srv1776039349:84 , srv1871522006:84 , srv1891518889:84 , srv1944536138:84 , srv1965204166:84 , srv2025734038:84 , srv2039364009:84 , srv2069401623:84 , srv209714856:84 , srv2138811599:84 , srv2143585644:84 , srv291246251:84 , srv299344076:84 , srv306430723:84 , srv34377132:84 , srv365579606:84 , srv381924643:84 , srv391085017:84 , srv407341666:84 , srv414887426:84 , srv467883906:84 , srv512721246:84 , srv515549340:84 , srv525756889:84 , srv526623844:84 , srv527175959:84 , srv530240784:84 , srv553745420:84 , srv557957781:84 , srv571516055:84 , srv577504057:84 , srv580819471:84 , srv617158084:84 , srv62731174:84 , srv628792546:84 , srv670751110:84 , srv6925371:84 , srv699268918:84 , srv713608329:84 , srv746759393:84 , srv7671403:84 , srv767582430:84 , srv769007821:84 , srv773798363:84 , srv78550135:84 , srv840672661:84 , srv843980445:84 , srv885484897:84 , srv920156928:84 , srv922823792:84 , srv965134922:84 , srv966279153:84 , srv974746182:84 , srv1251066494:85 , srv1623654077:85 , srv1664425027:85 , srv201361064:85 , srv2070103440:85 , srv315407227:85 , srv385764542:85 , srv97166646:85 , srv254818335:684 }
2024-03-15 18:53:28,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-15 18:53:28,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv670751110=80, srv2039364009=44, srv843980445=92, srv254818335=50, srv1669131229=31, srv1715097040=32, srv467883906=63, srv1463124868=16, srv1273238834=7, srv769007821=87, srv1769472022=35, srv798037448=90, srv974746182=99, srv965134922=96, srv1635286883=28, srv1251066494=6, srv557957781=72, srv1434800321=12, srv97166646=98, srv1623632017=25, srv7671403=85, srv553745420=71, srv315407227=54, srv1756271896=34, srv299344076=52, srv1439966720=13, srv512721246=65, srv391085017=60, srv2069401623=45, srv508685881=64, srv1042963007=0, srv1509430142=20, srv1944536138=39, srv773798363=88, srv201361064=42, srv966279153=97, srv1594606193=23, srv407341666=61, srv525756889=67, srv1199383193=4, srv34377132=55, srv1776039349=36, srv840672661=91, srv1664425027=30, srv6925371=81, srv1659990052=29, srv1473840153=17, srv1380898481=11, srv385764542=58, srv1286297603=8, srv291246251=51, srv1451476386=15, srv628792546=79, srv577504057=74, srv1623654077=26, srv209714856=47, srv515549340=66, srv306430723=53, srv767582430=86, srv199346244=41, srv1581364861=22, srv59443007=76, srv2143585644=49, srv530240784=70, srv571516055=73, srv1238300576=5, srv580819471=75, srv1871522006=37, srv1630690221=27, srv2138811599=48, srv2070103440=46, srv62731174=78, srv2025734038=43, srv1504306255=19, srv1595824932=24, srv1322619401=9, srv1891518889=38, srv527175959=69, srv885484897=93, srv1379367633=10, srv920156928=94, srv922823792=95, srv699268918=82, srv746759393=84, srv1722847927=33, srv414887426=62, srv1450166210=14, srv38603165=59, srv1097848214=1, srv153146935=21, srv526623844=68, srv713608329=83, srv78550135=89, srv1185962669=2, srv1198341721=3, srv1499284603=18, srv617158084=77, srv365579606=56, srv381924643=57, srv1965204166=40} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-15 18:53:28,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-15 18:53:28,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-15 18:53:28,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-15 18:53:28,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-15 18:53:28,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-15 18:53:28,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-15 18:53:28,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-15 18:53:28,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-15 18:53:28,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-15 18:53:28,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-15 18:53:28,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-15 18:53:28,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-15 18:53:28,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-15 18:53:28,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-15 18:53:28,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-15 18:53:28,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-15 18:53:28,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-15 18:53:28,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-15 18:53:28,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-15 18:53:28,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-15 18:53:28,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-15 18:53:28,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-15 18:53:28,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-15 18:53:28,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-15 18:53:28,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-15 18:53:28,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-15 18:53:28,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-15 18:53:28,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-15 18:53:28,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-15 18:53:28,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-15 18:53:28,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-15 18:53:28,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-15 18:53:28,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-15 18:53:28,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-15 18:53:28,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-15 18:53:28,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-15 18:53:28,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-15 18:53:28,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-15 18:53:28,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-15 18:53:28,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-15 18:53:28,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-15 18:53:28,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-15 18:53:28,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-15 18:53:28,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-15 18:53:28,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-15 18:53:28,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-15 18:53:28,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-15 18:53:28,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-15 18:53:28,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-15 18:53:28,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-15 18:53:28,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-15 18:53:28,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-15 18:53:28,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-15 18:53:28,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-15 18:53:28,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-15 18:53:28,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-15 18:53:28,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-15 18:53:28,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-15 18:53:28,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-15 18:53:28,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-15 18:53:28,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-15 18:53:28,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-15 18:53:28,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-15 18:53:28,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-15 18:53:28,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-15 18:53:28,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-15 18:53:28,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-15 18:53:28,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-15 18:53:28,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-15 18:53:28,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-15 18:53:28,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-15 18:53:28,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-15 18:53:28,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-15 18:53:28,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-15 18:53:28,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-15 18:53:28,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-15 18:53:28,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-15 18:53:28,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-15 18:53:28,909 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-15 18:53:29,023 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=5.223933426232788E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0011605177063713189); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-15 18:53:29,024 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1273238834:83 , srv1509430142:83 , srv1594606193:83 , srv199346244:83 , srv38603165:83 , srv508685881:83 , srv59443007:83 , srv798037448:83 , srv1042963007:84 , srv1097848214:84 , srv1185962669:84 , srv1198341721:84 , srv1199383193:84 , srv1238300576:84 , srv1286297603:84 , srv1322619401:84 , srv1379367633:84 , srv1380898481:84 , srv1434800321:84 , srv1439966720:84 , srv1450166210:84 , srv1451476386:84 , srv1463124868:84 , srv1473840153:84 , srv1499284603:84 , srv1504306255:84 , srv153146935:84 , srv1581364861:84 , srv1595824932:84 , srv1623632017:84 , srv1630690221:84 , srv1635286883:84 , srv1659990052:84 , srv1669131229:84 , srv1715097040:84 , srv1722847927:84 , srv1756271896:84 , srv1769472022:84 , srv1776039349:84 , srv1871522006:84 , srv1891518889:84 , srv1944536138:84 , srv1965204166:84 , srv2025734038:84 , srv2039364009:84 , srv2069401623:84 , srv209714856:84 , srv2138811599:84 , srv2143585644:84 , srv291246251:84 , srv299344076:84 , srv306430723:84 , srv34377132:84 , srv365579606:84 , srv381924643:84 , srv391085017:84 , srv407341666:84 , srv414887426:84 , srv467883906:84 , srv512721246:84 , srv515549340:84 , srv525756889:84 , srv526623844:84 , srv527175959:84 , srv530240784:84 , srv553745420:84 , srv557957781:84 , srv571516055:84 , srv577504057:84 , srv580819471:84 , srv617158084:84 , srv62731174:84 , srv628792546:84 , srv670751110:84 , srv6925371:84 , srv699268918:84 , srv713608329:84 , srv746759393:84 , srv7671403:84 , srv767582430:84 , srv769007821:84 , srv773798363:84 , srv78550135:84 , srv840672661:84 , srv843980445:84 , srv885484897:84 , srv920156928:84 , srv922823792:84 , srv965134922:84 , srv966279153:84 , srv974746182:84 , srv1251066494:85 , srv1623654077:85 , srv1664425027:85 , srv201361064:85 , srv2070103440:85 , srv315407227:85 , srv385764542:85 , srv97166646:85 , srv254818335:684 }
2024-03-15 18:53:29,048 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=225 (was 225), ProcessCount=170 (was 170), AvailableMemoryMB=4166 (was 4668)
2024-03-15 18:53:29,058 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=225, ProcessCount=170, AvailableMemoryMB=4165
2024-03-15 18:53:29,059 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-15 18:53:29,059 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-03-15 18:53:29,067 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1055257478:84 , srv1060375341:84 , srv1103418555:84 , srv1145217837:84 , srv1332684337:84 , srv1415830838:84 , srv1416409197:84 , srv145216901:84 , srv1563078485:84 , srv161456302:84 , srv1634650188:84 , srv1693185109:84 , srv1747822347:84 , srv189866000:84 , srv1980259488:84 , srv198375038:84 , srv1989980152:84 , srv2042750642:84 , srv260525086:84 , srv279671884:84 , srv30275857:84 , srv484298430:84 , srv514461022:84 , srv518931116:84 , srv6020402:84 , srv853616497:84 , srv890891287:84 , srv943967115:84 , srv977251497:84 , srv134175307:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-03-15 18:53:29,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-15 18:53:29,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv484298430=22, srv145216901=8, srv1980259488=15, srv2042750642=18, srv1416409197=7, srv943967115=28, srv1055257478=0, srv1145217837=3, srv890891287=27, srv1060375341=1, srv1563078485=9, srv1634650188=11, srv260525086=19, srv1103418555=2, srv1747822347=13, srv977251497=29, srv198375038=16, srv1332684337=4, srv279671884=20, srv1415830838=6, srv30275857=21, srv189866000=14, srv1693185109=12, srv518931116=24, srv514461022=23, srv853616497=26, srv134175307=5, srv6020402=25, srv161456302=10, srv1989980152=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-15 18:53:29,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-15 18:53:29,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-15 18:53:29,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-15 18:53:29,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-15 18:53:29,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-15 18:53:29,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-15 18:53:29,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-15 18:53:29,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-15 18:53:29,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-15 18:53:29,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-15 18:53:29,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-15 18:53:29,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-15 18:53:29,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-15 18:53:29,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-15 18:53:29,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-15 18:53:29,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-15 18:53:29,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-15 18:53:29,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-15 18:53:29,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-15 18:53:29,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-15 18:53:29,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-15 18:53:29,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-15 18:53:29,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-15 18:53:29,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-15 18:53:29,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-15 18:53:29,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-15 18:53:29,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-15 18:53:29,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-15 18:53:29,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-15 18:53:29,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-15 18:53:29,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-15 18:53:29,095 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1124930282548966, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-15 18:53:29,095 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912547505123076, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1124930282548966, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-03-15 18:55:29,085 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5840800 different iterations.  Found a solution that moves 1853 regions; Going from a computed imbalance of 0.9912547505123076 to a new imbalance of 4.324994439959734E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6862962962962963, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-15 18:55:29,093 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1055257478:90 , srv1060375341:90 , srv1103418555:90 , srv1145217837:90 , srv1332684337:90 , srv134175307:90 , srv1415830838:90 , srv1416409197:90 , srv145216901:90 , srv1563078485:90 , srv161456302:90 , srv1634650188:90 , srv1693185109:90 , srv1747822347:90 , srv189866000:90 , srv1980259488:90 , srv198375038:90 , srv1989980152:90 , srv2042750642:90 , srv260525086:90 , srv279671884:90 , srv30275857:90 , srv484298430:90 , srv514461022:90 , srv518931116:90 , srv6020402:90 , srv853616497:90 , srv890891287:90 , srv943967115:90 , srv977251497:90 }
2024-03-15 18:55:29,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-15 18:55:29,095 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv484298430=22, srv145216901=8, srv1980259488=15, srv2042750642=18, srv1416409197=7, srv943967115=28, srv1055257478=0, srv1145217837=3, srv890891287=27, srv1060375341=1, srv1563078485=9, srv1634650188=11, srv260525086=19, srv1103418555=2, srv1747822347=13, srv977251497=29, srv198375038=16, srv1332684337=4, srv279671884=20, srv1415830838=6, srv30275857=21, srv189866000=14, srv1693185109=12, srv518931116=24, srv514461022=23, srv853616497=26, srv134175307=5, srv6020402=25, srv161456302=10, srv1989980152=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-15 18:55:29,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-15 18:55:29,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-15 18:55:29,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-15 18:55:29,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-15 18:55:29,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-15 18:55:29,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-15 18:55:29,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-15 18:55:29,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-15 18:55:29,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-15 18:55:29,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-15 18:55:29,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-15 18:55:29,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-15 18:55:29,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-15 18:55:29,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-15 18:55:29,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-15 18:55:29,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-15 18:55:29,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-15 18:55:29,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-15 18:55:29,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-15 18:55:29,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-15 18:55:29,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-15 18:55:29,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-15 18:55:29,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-15 18:55:29,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-15 18:55:29,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-15 18:55:29,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-15 18:55:29,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-15 18:55:29,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-15 18:55:29,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-15 18:55:29,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-15 18:55:29,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-15 18:55:29,121 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-15 18:55:29,121 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1055257478:90 , srv1060375341:90 , srv1103418555:90 , srv1145217837:90 , srv1332684337:90 , srv134175307:90 , srv1415830838:90 , srv1416409197:90 , srv145216901:90 , srv1563078485:90 , srv161456302:90 , srv1634650188:90 , srv1693185109:90 , srv1747822347:90 , srv189866000:90 , srv1980259488:90 , srv198375038:90 , srv1989980152:90 , srv2042750642:90 , srv260525086:90 , srv279671884:90 , srv30275857:90 , srv484298430:90 , srv514461022:90 , srv518931116:90 , srv6020402:90 , srv853616497:90 , srv890891287:90 , srv943967115:90 , srv977251497:90 }
2024-03-15 18:55:29,136 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=157 (was 225), ProcessCount=168 (was 170), AvailableMemoryMB=3367 (was 4165)