2024-01-28 22:53:17,224 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-01-28 22:53:17,537 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-01-28 22:53:17,663 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-01-28 22:53:17,731 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-28 22:53:17,738 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-01-28 22:53:17,752 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=285, ProcessCount=167, AvailableMemoryMB=3388
2024-01-28 22:53:17,755 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-28 22:53:17,755 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-01-28 22:53:17,896 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1010874062:84 , srv1035276344:84 , srv1043992105:84 , srv105219275:84 , srv106508637:84 , srv1072906082:84 , srv1076181606:84 , srv1081755690:84 , srv1094536021:84 , srv113028445:84 , srv1138700888:84 , srv1142045055:84 , srv1170163738:84 , srv1184211421:84 , srv1187345432:84 , srv1194317391:84 , srv1207833815:84 , srv1208866723:84 , srv1246219718:84 , srv125551427:84 , srv1261025374:84 , srv1262678352:84 , srv1269722399:84 , srv1279319560:84 , srv1336769151:84 , srv1365271214:84 , srv137542090:84 , srv1379481063:84 , srv1399545454:84 , srv1410539506:84 , srv1423010703:84 , srv1423318811:84 , srv144785900:84 , srv1456537538:84 , srv1520119812:84 , srv1557899073:84 , srv1625454769:84 , srv1636995353:84 , srv1641877846:84 , srv1667961107:84 , srv1669134071:84 , srv1691512634:84 , srv1699429575:84 , srv175017347:84 , srv1767750751:84 , srv1768998369:84 , srv1789751244:84 , srv1795939254:84 , srv1834719377:84 , srv1837996624:84 , srv1839197841:84 , srv1870530017:84 , srv1941673759:84 , srv1946111224:84 , srv1950504933:84 , srv1987047779:84 , srv1990372834:84 , srv2036826827:84 , srv2039564257:84 , srv2083631045:84 , srv2089077927:84 , srv2114182525:84 , srv2118112781:84 , srv2137399537:84 , srv2139088525:84 , srv2142252987:84 , srv218231011:84 , srv235065032:84 , srv251650302:84 , srv260188711:84 , srv267487181:84 , srv320435695:84 , srv355268967:84 , srv379973383:84 , srv380077402:84 , srv382751709:84 , srv390668236:84 , srv450749582:84 , srv452031461:84 , srv460958600:84 , srv496767162:84 , srv53155336:84 , srv595405446:84 , srv619881801:84 , srv661519157:84 , srv668320237:84 , srv685732687:84 , srv6907637:84 , srv746044696:84 , srv75142071:84 , srv841869434:84 , srv860530246:84 , srv863266028:84 , srv907474950:84 , srv916492033:84 , srv930243914:84 , srv973117251:84 , srv97383712:84 , srv990894234:84 , srv1029743639:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-01-28 22:53:17,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-28 22:53:17,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv175017347=44, srv930243914=96, srv1138700888=11, srv75142071=90, srv1187345432=15, srv1667961107=40, srv106508637=5, srv1365271214=26, srv990894234=99, srv1410539506=30, srv260188711=70, srv1207833815=17, srv1269722399=23, srv1795939254=48, srv382751709=76, srv1170163738=13, srv746044696=89, srv1184211421=14, srv137542090=27, srv1072906082=6, srv1208866723=18, srv1699429575=43, srv2039564257=59, srv6907637=88, srv97383712=98, srv390668236=77, srv1768998369=46, srv1456537538=34, srv1423010703=31, srv1557899073=36, srv355268967=73, srv661519157=85, srv267487181=71, srv105219275=4, srv452031461=79, srv1789751244=47, srv1423318811=32, srv235065032=68, srv2142252987=66, srv1043992105=3, srv113028445=10, srv2139088525=65, srv1669134071=41, srv841869434=91, srv1870530017=52, srv2036826827=58, srv1076181606=7, srv251650302=69, srv860530246=92, srv907474950=94, srv668320237=86, srv1029743639=1, srv1081755690=8, srv1520119812=35, srv1636995353=38, srv1262678352=22, srv1987047779=56, srv1625454769=37, srv53155336=82, srv1941673759=53, srv1279319560=24, srv460958600=80, srv619881801=84, srv2137399537=64, srv380077402=75, srv973117251=97, srv320435695=72, srv144785900=33, srv2114182525=62, srv1194317391=16, srv1035276344=2, srv1379481063=28, srv1946111224=54, srv1142045055=12, srv1399545454=29, srv218231011=67, srv2118112781=63, srv1094536021=9, srv1010874062=0, srv125551427=20, srv1837996624=50, srv863266028=93, srv1950504933=55, srv379973383=74, srv1990372834=57, srv685732687=87, srv1336769151=25, srv1834719377=49, srv1246219718=19, srv1641877846=39, srv595405446=83, srv1839197841=51, srv1767750751=45, srv496767162=81, srv450749582=78, srv1261025374=21, srv2083631045=60, srv916492033=95, srv2089077927=61, srv1691512634=42} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-28 22:53:18,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-28 22:53:18,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-28 22:53:18,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-28 22:53:18,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-28 22:53:18,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-28 22:53:18,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-28 22:53:18,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-28 22:53:18,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-28 22:53:18,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-28 22:53:18,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-28 22:53:18,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-28 22:53:18,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-28 22:53:18,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-28 22:53:18,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-28 22:53:18,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-28 22:53:18,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-28 22:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-28 22:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-28 22:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-28 22:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-28 22:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-28 22:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-28 22:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-28 22:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-28 22:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-28 22:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-28 22:53:18,024 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-28 22:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-28 22:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-28 22:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-28 22:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-28 22:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-28 22:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-28 22:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-28 22:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-28 22:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-28 22:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-28 22:53:18,025 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-28 22:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-28 22:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-28 22:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-28 22:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-28 22:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-28 22:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-28 22:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-28 22:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-28 22:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-28 22:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-28 22:53:18,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-28 22:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-28 22:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-28 22:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-28 22:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-28 22:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-28 22:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-28 22:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-28 22:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-28 22:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-28 22:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-28 22:53:18,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-28 22:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-28 22:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-28 22:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-28 22:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-28 22:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-28 22:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-28 22:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-28 22:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-28 22:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-28 22:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-28 22:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-28 22:53:18,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-28 22:53:18,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-28 22:53:18,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-28 22:53:18,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-28 22:53:18,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-28 22:53:18,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-28 22:53:18,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-28 22:53:18,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-28 22:53:18,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-28 22:53:18,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-28 22:53:18,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-28 22:53:18,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-28 22:53:18,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-28 22:53:18,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-28 22:53:18,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-28 22:53:18,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-28 22:53:18,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-28 22:53:18,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-28 22:53:18,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-28 22:53:18,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-28 22:53:18,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-28 22:53:18,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-28 22:53:18,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-28 22:53:18,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-28 22:53:18,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-28 22:53:18,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-28 22:53:18,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-28 22:53:18,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-28 22:53:18,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-28 22:53:18,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-28 22:53:18,241 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-28 22:53:18,242 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-28 22:53:18,242 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-28 22:53:18,645 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 587 ms to try 5000 different iterations.  Found a solution that moves 4671 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18238470465114495. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0390305261023524, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05512968080760465, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.519, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0390305261023524, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17733333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24733333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-28 22:53:18,659 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1262678352:82 , srv144785900:83 , srv1767750751:83 , srv355268967:83 , srv746044696:83 , srv1043992105:84 , srv1076181606:84 , srv113028445:84 , srv1142045055:84 , srv125551427:84 , srv1261025374:84 , srv1768998369:84 , srv1789751244:84 , srv1946111224:84 , srv2083631045:84 , srv2139088525:84 , srv382751709:84 , srv907474950:84 , srv1010874062:85 , srv106508637:85 , srv1072906082:85 , srv1184211421:85 , srv1194317391:85 , srv1208866723:85 , srv1410539506:85 , srv1456537538:85 , srv2142252987:85 , srv267487181:85 , srv380077402:85 , srv53155336:85 , srv619881801:85 , srv661519157:85 , srv685732687:85 , srv863266028:85 , srv973117251:85 , srv990894234:85 , srv1035276344:86 , srv105219275:86 , srv1081755690:86 , srv1269722399:86 , srv1279319560:86 , srv1365271214:86 , srv1379481063:86 , srv1423010703:86 , srv1423318811:86 , srv1520119812:86 , srv1667961107:86 , srv1669134071:86 , srv1699429575:86 , srv1834719377:86 , srv1870530017:86 , srv1990372834:86 , srv2036826827:86 , srv2039564257:86 , srv2118112781:86 , srv218231011:86 , srv320435695:86 , srv450749582:86 , srv841869434:86 , srv97383712:86 , srv1094536021:87 , srv1170163738:87 , srv1187345432:87 , srv137542090:87 , srv1557899073:87 , srv1625454769:87 , srv1795939254:87 , srv1941673759:87 , srv1950504933:87 , srv1987047779:87 , srv2137399537:87 , srv496767162:87 , srv860530246:87 , srv916492033:87 , srv930243914:87 , srv1138700888:88 , srv260188711:88 , srv379973383:88 , srv452031461:88 , srv460958600:88 , srv668320237:88 , srv1399545454:89 , srv175017347:89 , srv2114182525:89 , srv6907637:89 , srv75142071:89 , srv1336769151:90 , srv1636995353:90 , srv595405446:90 , srv1691512634:91 , srv1837996624:91 , srv1839197841:91 , srv235065032:91 , srv390668236:91 , srv1207833815:92 , srv1246219718:92 , srv1641877846:92 , srv2089077927:92 , srv251650302:92 , srv1029743639:437 }
2024-01-28 22:53:18,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-28 22:53:18,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv175017347=44, srv930243914=96, srv1138700888=11, srv75142071=90, srv1187345432=15, srv1667961107=40, srv106508637=5, srv1365271214=26, srv990894234=99, srv1410539506=30, srv260188711=70, srv1207833815=17, srv1269722399=23, srv1795939254=48, srv382751709=76, srv1170163738=13, srv746044696=89, srv1184211421=14, srv137542090=27, srv1072906082=6, srv1208866723=18, srv1699429575=43, srv2039564257=59, srv6907637=88, srv97383712=98, srv390668236=77, srv1768998369=46, srv1456537538=34, srv1423010703=31, srv1557899073=36, srv355268967=73, srv661519157=85, srv267487181=71, srv105219275=4, srv452031461=79, srv1789751244=47, srv1423318811=32, srv235065032=68, srv2142252987=66, srv1043992105=3, srv113028445=10, srv2139088525=65, srv1669134071=41, srv841869434=91, srv1870530017=52, srv2036826827=58, srv1076181606=7, srv251650302=69, srv860530246=92, srv907474950=94, srv668320237=86, srv1029743639=1, srv1081755690=8, srv1520119812=35, srv1636995353=38, srv1262678352=22, srv1987047779=56, srv1625454769=37, srv53155336=82, srv1941673759=53, srv1279319560=24, srv460958600=80, srv619881801=84, srv2137399537=64, srv380077402=75, srv973117251=97, srv320435695=72, srv144785900=33, srv2114182525=62, srv1194317391=16, srv1035276344=2, srv1379481063=28, srv1946111224=54, srv1142045055=12, srv1399545454=29, srv218231011=67, srv2118112781=63, srv1094536021=9, srv1010874062=0, srv125551427=20, srv1837996624=50, srv863266028=93, srv1950504933=55, srv379973383=74, srv1990372834=57, srv685732687=87, srv1336769151=25, srv1834719377=49, srv1246219718=19, srv1641877846=39, srv595405446=83, srv1839197841=51, srv1767750751=45, srv496767162=81, srv450749582=78, srv1261025374=21, srv2083631045=60, srv916492033=95, srv2089077927=61, srv1691512634=42} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-28 22:53:18,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-28 22:53:18,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-28 22:53:18,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-28 22:53:18,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-28 22:53:18,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-28 22:53:18,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-28 22:53:18,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-28 22:53:18,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-28 22:53:18,689 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-28 22:53:18,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-28 22:53:18,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-28 22:53:18,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-28 22:53:18,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-28 22:53:18,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-28 22:53:18,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-28 22:53:18,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-28 22:53:18,690 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-28 22:53:18,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-28 22:53:18,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-28 22:53:18,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-28 22:53:18,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-28 22:53:18,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-28 22:53:18,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-28 22:53:18,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-28 22:53:18,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-28 22:53:18,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-28 22:53:18,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-28 22:53:18,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-28 22:53:18,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-28 22:53:18,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-28 22:53:18,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-28 22:53:18,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-28 22:53:18,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-28 22:53:18,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-28 22:53:18,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-28 22:53:18,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-28 22:53:18,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-28 22:53:18,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-28 22:53:18,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-28 22:53:18,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-28 22:53:18,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-28 22:53:18,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-28 22:53:18,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-28 22:53:18,693 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-28 22:53:18,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-28 22:53:18,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-28 22:53:18,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-28 22:53:18,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-28 22:53:18,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-28 22:53:18,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-28 22:53:18,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-28 22:53:18,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-28 22:53:18,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-28 22:53:18,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-28 22:53:18,694 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-28 22:53:18,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-28 22:53:18,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-28 22:53:18,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-28 22:53:18,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-28 22:53:18,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-28 22:53:18,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-28 22:53:18,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-28 22:53:18,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-28 22:53:18,695 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-28 22:53:18,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-28 22:53:18,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-28 22:53:18,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-28 22:53:18,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-28 22:53:18,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-28 22:53:18,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-28 22:53:18,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-28 22:53:18,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-28 22:53:18,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-28 22:53:18,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-28 22:53:18,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-28 22:53:18,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-28 22:53:18,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-28 22:53:18,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-28 22:53:18,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-28 22:53:18,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-28 22:53:18,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-28 22:53:18,697 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-28 22:53:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-28 22:53:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-28 22:53:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-28 22:53:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-28 22:53:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-28 22:53:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-28 22:53:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-28 22:53:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-28 22:53:18,698 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-28 22:53:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-28 22:53:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-28 22:53:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-28 22:53:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-28 22:53:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-28 22:53:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-28 22:53:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-28 22:53:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-28 22:53:18,699 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-28 22:53:18,700 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-28 22:53:18,859 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0390305261023524, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05512968080760465, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0390305261023524, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17733333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24733333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-28 22:53:18,860 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-28 22:53:18,860 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18235199761008333, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0390305261023524, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05512968080760465, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0390305261023524, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17733333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24733333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-28 22:53:19,127 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 397 ms to try 5000 different iterations.  Found a solution that moves 2626 regions; Going from a computed imbalance of 0.18235199761008333 to a new imbalance of 0.015727187810332895. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019045694948513412); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031060564999562248, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2917777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019045694948513412); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04525, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-28 22:53:19,139 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1207833815:78 , srv1246219718:78 , srv1641877846:81 , srv390668236:81 , srv595405446:81 , srv1691512634:82 , srv1839197841:82 , srv235065032:82 , srv251650302:82 , srv75142071:82 , srv930243914:82 , srv1336769151:83 , srv1399545454:83 , srv175017347:83 , srv2114182525:83 , srv260188711:83 , srv1081755690:84 , srv1625454769:84 , srv1795939254:84 , srv1837996624:84 , srv1987047779:84 , srv2089077927:84 , srv2137399537:84 , srv452031461:84 , srv1184211421:85 , srv1187345432:85 , srv1194317391:85 , srv1269722399:85 , srv1365271214:85 , srv1520119812:85 , srv1557899073:85 , srv1667961107:85 , srv1941673759:85 , srv1990372834:85 , srv2039564257:85 , srv2142252987:85 , srv379973383:85 , srv450749582:85 , srv460958600:85 , srv841869434:85 , srv1010874062:86 , srv105219275:86 , srv1072906082:86 , srv113028445:86 , srv1170163738:86 , srv1208866723:86 , srv125551427:86 , srv1423318811:86 , srv1636995353:86 , srv2118112781:86 , srv267487181:86 , srv496767162:86 , srv619881801:86 , srv661519157:86 , srv668320237:86 , srv685732687:86 , srv6907637:86 , srv916492033:86 , srv973117251:86 , srv97383712:86 , srv1094536021:87 , srv1279319560:87 , srv137542090:87 , srv1379481063:87 , srv1423010703:87 , srv144785900:87 , srv1669134071:87 , srv1768998369:87 , srv1870530017:87 , srv1946111224:87 , srv1950504933:87 , srv2036826827:87 , srv2083631045:87 , srv218231011:87 , srv320435695:87 , srv382751709:87 , srv860530246:87 , srv1035276344:88 , srv1043992105:88 , srv106508637:88 , srv1138700888:88 , srv1699429575:88 , srv380077402:88 , srv53155336:88 , srv863266028:88 , srv907474950:88 , srv990894234:88 , srv1142045055:89 , srv1261025374:89 , srv1410539506:89 , srv1789751244:89 , srv1834719377:89 , srv746044696:89 , srv1076181606:90 , srv1262678352:90 , srv2139088525:90 , srv355268967:90 , srv1767750751:91 , srv1456537538:92 , srv1029743639:506 }
2024-01-28 22:53:19,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-28 22:53:19,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv175017347=44, srv930243914=96, srv1138700888=11, srv75142071=90, srv1187345432=15, srv1667961107=40, srv106508637=5, srv1365271214=26, srv990894234=99, srv1410539506=30, srv260188711=70, srv1207833815=17, srv1269722399=23, srv1795939254=48, srv382751709=76, srv1170163738=13, srv746044696=89, srv1184211421=14, srv137542090=27, srv1072906082=6, srv1208866723=18, srv1699429575=43, srv2039564257=59, srv6907637=88, srv97383712=98, srv390668236=77, srv1768998369=46, srv1456537538=34, srv1423010703=31, srv1557899073=36, srv355268967=73, srv661519157=85, srv267487181=71, srv105219275=4, srv452031461=79, srv1789751244=47, srv1423318811=32, srv235065032=68, srv2142252987=66, srv1043992105=3, srv113028445=10, srv2139088525=65, srv1669134071=41, srv841869434=91, srv1870530017=52, srv2036826827=58, srv1076181606=7, srv251650302=69, srv860530246=92, srv907474950=94, srv668320237=86, srv1029743639=1, srv1081755690=8, srv1520119812=35, srv1636995353=38, srv1262678352=22, srv1987047779=56, srv1625454769=37, srv53155336=82, srv1941673759=53, srv1279319560=24, srv460958600=80, srv619881801=84, srv2137399537=64, srv380077402=75, srv973117251=97, srv320435695=72, srv144785900=33, srv2114182525=62, srv1194317391=16, srv1035276344=2, srv1379481063=28, srv1946111224=54, srv1142045055=12, srv1399545454=29, srv218231011=67, srv2118112781=63, srv1094536021=9, srv1010874062=0, srv125551427=20, srv1837996624=50, srv863266028=93, srv1950504933=55, srv379973383=74, srv1990372834=57, srv685732687=87, srv1336769151=25, srv1834719377=49, srv1246219718=19, srv1641877846=39, srv595405446=83, srv1839197841=51, srv1767750751=45, srv496767162=81, srv450749582=78, srv1261025374=21, srv2083631045=60, srv916492033=95, srv2089077927=61, srv1691512634=42} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-28 22:53:19,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-28 22:53:19,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-28 22:53:19,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-28 22:53:19,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-28 22:53:19,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-28 22:53:19,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-28 22:53:19,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-28 22:53:19,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-28 22:53:19,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-28 22:53:19,185 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-28 22:53:19,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-28 22:53:19,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-28 22:53:19,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-28 22:53:19,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-28 22:53:19,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-28 22:53:19,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-28 22:53:19,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-28 22:53:19,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-28 22:53:19,186 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-28 22:53:19,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-28 22:53:19,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-28 22:53:19,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-28 22:53:19,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-28 22:53:19,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-28 22:53:19,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-28 22:53:19,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-28 22:53:19,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-28 22:53:19,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-28 22:53:19,187 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-28 22:53:19,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-28 22:53:19,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-28 22:53:19,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-28 22:53:19,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-28 22:53:19,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-28 22:53:19,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-28 22:53:19,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-28 22:53:19,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-28 22:53:19,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-28 22:53:19,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-28 22:53:19,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-28 22:53:19,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-28 22:53:19,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-28 22:53:19,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-28 22:53:19,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-28 22:53:19,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-28 22:53:19,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-28 22:53:19,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-28 22:53:19,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-28 22:53:19,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-28 22:53:19,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-28 22:53:19,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-28 22:53:19,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-28 22:53:19,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-28 22:53:19,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-28 22:53:19,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-28 22:53:19,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-28 22:53:19,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-28 22:53:19,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-28 22:53:19,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-28 22:53:19,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-28 22:53:19,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-28 22:53:19,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-28 22:53:19,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-28 22:53:19,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-28 22:53:19,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-28 22:53:19,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-28 22:53:19,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-28 22:53:19,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-28 22:53:19,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-28 22:53:19,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-28 22:53:19,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-28 22:53:19,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-28 22:53:19,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-28 22:53:19,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-28 22:53:19,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-28 22:53:19,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-28 22:53:19,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-28 22:53:19,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-28 22:53:19,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-28 22:53:19,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-28 22:53:19,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-28 22:53:19,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-28 22:53:19,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-28 22:53:19,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-28 22:53:19,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-28 22:53:19,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-28 22:53:19,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-28 22:53:19,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-28 22:53:19,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-28 22:53:19,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-28 22:53:19,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-28 22:53:19,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-28 22:53:19,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-28 22:53:19,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-28 22:53:19,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-28 22:53:19,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-28 22:53:19,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-28 22:53:19,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-28 22:53:19,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-28 22:53:19,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-28 22:53:19,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-28 22:53:19,343 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019045694948513412); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031060564999562248, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019045694948513412); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04525, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-28 22:53:19,343 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-28 22:53:19,344 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015708800165325876, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019045694948513412); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031060564999562248, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019045694948513412); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012666666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04525, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-28 22:53:19,581 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 363 ms to try 5000 different iterations.  Found a solution that moves 980 regions; Going from a computed imbalance of 0.015708800165325876 to a new imbalance of 6.755560450693517E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00454158708147337); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0053602016813824645); MoveCostFunction : (multiplier=7.0, imbalance=0.10888888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00454158708147337); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-28 22:53:19,583 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1138700888:81 , srv1456537538:81 , srv1636995353:81 , srv1834719377:82 , srv2089077927:82 , srv1035276344:83 , srv1410539506:83 , srv1699429575:83 , srv1837996624:83 , srv1950504933:83 , srv2139088525:83 , srv251650302:83 , srv668320237:83 , srv6907637:83 , srv1076181606:84 , srv1094536021:84 , srv1142045055:84 , srv1170163738:84 , srv1336769151:84 , srv137542090:84 , srv1641877846:84 , srv1669134071:84 , srv1767750751:84 , srv1839197841:84 , srv379973383:84 , srv460958600:84 , srv1043992105:85 , srv105219275:85 , srv106508637:85 , srv1261025374:85 , srv1262678352:85 , srv1279319560:85 , srv1379481063:85 , srv1399545454:85 , srv1423010703:85 , srv1557899073:85 , srv1691512634:85 , srv1768998369:85 , srv1789751244:85 , srv1870530017:85 , srv2114182525:85 , srv2118112781:85 , srv218231011:85 , srv235065032:85 , srv320435695:85 , srv355268967:85 , srv380077402:85 , srv496767162:85 , srv53155336:85 , srv860530246:85 , srv907474950:85 , srv97383712:85 , srv990894234:85 , srv1072906082:86 , srv1187345432:86 , srv1269722399:86 , srv1625454769:86 , srv175017347:86 , srv1941673759:86 , srv1946111224:86 , srv2036826827:86 , srv2039564257:86 , srv260188711:86 , srv390668236:86 , srv452031461:86 , srv595405446:86 , srv685732687:86 , srv746044696:86 , srv75142071:86 , srv841869434:86 , srv863266028:86 , srv916492033:86 , srv973117251:86 , srv1010874062:87 , srv1081755690:87 , srv113028445:87 , srv1194317391:87 , srv1208866723:87 , srv125551427:87 , srv1423318811:87 , srv1520119812:87 , srv1667961107:87 , srv1795939254:87 , srv1987047779:87 , srv1990372834:87 , srv2083631045:87 , srv2137399537:87 , srv2142252987:87 , srv267487181:87 , srv382751709:87 , srv450749582:87 , srv619881801:87 , srv661519157:87 , srv1184211421:88 , srv1207833815:88 , srv1246219718:88 , srv1365271214:88 , srv144785900:88 , srv930243914:88 , srv1029743639:555 }
2024-01-28 22:53:19,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-28 22:53:19,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv175017347=44, srv930243914=96, srv1138700888=11, srv75142071=90, srv1187345432=15, srv1667961107=40, srv106508637=5, srv1365271214=26, srv990894234=99, srv1410539506=30, srv260188711=70, srv1207833815=17, srv1269722399=23, srv1795939254=48, srv382751709=76, srv1170163738=13, srv746044696=89, srv1184211421=14, srv137542090=27, srv1072906082=6, srv1208866723=18, srv1699429575=43, srv2039564257=59, srv6907637=88, srv97383712=98, srv390668236=77, srv1768998369=46, srv1456537538=34, srv1423010703=31, srv1557899073=36, srv355268967=73, srv661519157=85, srv267487181=71, srv105219275=4, srv452031461=79, srv1789751244=47, srv1423318811=32, srv235065032=68, srv2142252987=66, srv1043992105=3, srv113028445=10, srv2139088525=65, srv1669134071=41, srv841869434=91, srv1870530017=52, srv2036826827=58, srv1076181606=7, srv251650302=69, srv860530246=92, srv907474950=94, srv668320237=86, srv1029743639=1, srv1081755690=8, srv1520119812=35, srv1636995353=38, srv1262678352=22, srv1987047779=56, srv1625454769=37, srv53155336=82, srv1941673759=53, srv1279319560=24, srv460958600=80, srv619881801=84, srv2137399537=64, srv380077402=75, srv973117251=97, srv320435695=72, srv144785900=33, srv2114182525=62, srv1194317391=16, srv1035276344=2, srv1379481063=28, srv1946111224=54, srv1142045055=12, srv1399545454=29, srv218231011=67, srv2118112781=63, srv1094536021=9, srv1010874062=0, srv125551427=20, srv1837996624=50, srv863266028=93, srv1950504933=55, srv379973383=74, srv1990372834=57, srv685732687=87, srv1336769151=25, srv1834719377=49, srv1246219718=19, srv1641877846=39, srv595405446=83, srv1839197841=51, srv1767750751=45, srv496767162=81, srv450749582=78, srv1261025374=21, srv2083631045=60, srv916492033=95, srv2089077927=61, srv1691512634=42} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-28 22:53:19,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-28 22:53:19,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-28 22:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-28 22:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-28 22:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-28 22:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-28 22:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-28 22:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-28 22:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-28 22:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-28 22:53:19,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-28 22:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-28 22:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-28 22:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-28 22:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-28 22:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-28 22:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-28 22:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-28 22:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-28 22:53:19,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-28 22:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-28 22:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-28 22:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-28 22:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-28 22:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-28 22:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-28 22:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-28 22:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-28 22:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-28 22:53:19,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-28 22:53:19,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-28 22:53:19,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-28 22:53:19,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-28 22:53:19,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-28 22:53:19,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-28 22:53:19,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-28 22:53:19,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-28 22:53:19,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-28 22:53:19,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-28 22:53:19,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-28 22:53:19,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-28 22:53:19,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-28 22:53:19,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-28 22:53:19,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-28 22:53:19,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-28 22:53:19,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-28 22:53:19,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-28 22:53:19,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-28 22:53:19,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-28 22:53:19,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-28 22:53:19,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-28 22:53:19,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-28 22:53:19,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-28 22:53:19,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-28 22:53:19,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-28 22:53:19,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-28 22:53:19,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-28 22:53:19,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-28 22:53:19,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-28 22:53:19,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-28 22:53:19,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-28 22:53:19,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-28 22:53:19,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-28 22:53:19,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-28 22:53:19,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-28 22:53:19,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-28 22:53:19,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-28 22:53:19,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-28 22:53:19,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-28 22:53:19,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-28 22:53:19,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-28 22:53:19,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-28 22:53:19,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-28 22:53:19,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-28 22:53:19,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-28 22:53:19,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-28 22:53:19,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-28 22:53:19,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-28 22:53:19,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-28 22:53:19,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-28 22:53:19,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-28 22:53:19,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-28 22:53:19,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-28 22:53:19,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-28 22:53:19,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-28 22:53:19,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-28 22:53:19,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-28 22:53:19,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-28 22:53:19,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-28 22:53:19,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-28 22:53:19,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-28 22:53:19,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-28 22:53:19,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-28 22:53:19,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-28 22:53:19,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-28 22:53:19,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-28 22:53:19,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-28 22:53:19,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-28 22:53:19,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-28 22:53:19,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-28 22:53:19,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-28 22:53:19,746 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00454158708147337); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0053602016813824645); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00454158708147337); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-28 22:53:19,746 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-28 22:53:19,746 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.68693938402605E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00454158708147337); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0053602016813824645); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00454158708147337); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-28 22:53:20,102 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 467 ms to try 5000 different iterations.  Found a solution that moves 216 regions; Going from a computed imbalance of 6.68693938402605E-4 to a new imbalance of 1.101345325899519E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.024); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-28 22:53:20,103 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1261025374:83 , srv1365271214:83 , srv1423318811:83 , srv2036826827:83 , srv390668236:83 , srv860530246:83 , srv863266028:83 , srv916492033:83 , srv1010874062:84 , srv106508637:84 , srv1076181606:84 , srv1094536021:84 , srv1184211421:84 , srv1187345432:84 , srv1207833815:84 , srv1208866723:84 , srv1246219718:84 , srv1279319560:84 , srv1336769151:84 , srv137542090:84 , srv1379481063:84 , srv1423010703:84 , srv144785900:84 , srv1456537538:84 , srv1520119812:84 , srv1667961107:84 , srv1691512634:84 , srv175017347:84 , srv1767750751:84 , srv1789751244:84 , srv1795939254:84 , srv1837996624:84 , srv1941673759:84 , srv1987047779:84 , srv1990372834:84 , srv2083631045:84 , srv2089077927:84 , srv2137399537:84 , srv218231011:84 , srv235065032:84 , srv267487181:84 , srv320435695:84 , srv355268967:84 , srv380077402:84 , srv382751709:84 , srv450749582:84 , srv496767162:84 , srv53155336:84 , srv619881801:84 , srv661519157:84 , srv6907637:84 , srv746044696:84 , srv990894234:84 , srv1035276344:85 , srv1043992105:85 , srv105219275:85 , srv1072906082:85 , srv113028445:85 , srv1138700888:85 , srv1170163738:85 , srv1194317391:85 , srv125551427:85 , srv1262678352:85 , srv1269722399:85 , srv1399545454:85 , srv1410539506:85 , srv1557899073:85 , srv1625454769:85 , srv1636995353:85 , srv1641877846:85 , srv1669134071:85 , srv1699429575:85 , srv1768998369:85 , srv1834719377:85 , srv1839197841:85 , srv1870530017:85 , srv1946111224:85 , srv1950504933:85 , srv2039564257:85 , srv2114182525:85 , srv2118112781:85 , srv2139088525:85 , srv2142252987:85 , srv251650302:85 , srv260188711:85 , srv379973383:85 , srv452031461:85 , srv460958600:85 , srv595405446:85 , srv668320237:85 , srv685732687:85 , srv75142071:85 , srv841869434:85 , srv907474950:85 , srv930243914:85 , srv973117251:85 , srv97383712:85 , srv1081755690:86 , srv1142045055:86 , srv1029743639:644 }
2024-01-28 22:53:20,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-28 22:53:20,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv175017347=44, srv930243914=96, srv1138700888=11, srv75142071=90, srv1187345432=15, srv1667961107=40, srv106508637=5, srv1365271214=26, srv990894234=99, srv1410539506=30, srv260188711=70, srv1207833815=17, srv1269722399=23, srv1795939254=48, srv382751709=76, srv1170163738=13, srv746044696=89, srv1184211421=14, srv137542090=27, srv1072906082=6, srv1208866723=18, srv1699429575=43, srv2039564257=59, srv6907637=88, srv97383712=98, srv390668236=77, srv1768998369=46, srv1456537538=34, srv1423010703=31, srv1557899073=36, srv355268967=73, srv661519157=85, srv267487181=71, srv105219275=4, srv452031461=79, srv1789751244=47, srv1423318811=32, srv235065032=68, srv2142252987=66, srv1043992105=3, srv113028445=10, srv2139088525=65, srv1669134071=41, srv841869434=91, srv1870530017=52, srv2036826827=58, srv1076181606=7, srv251650302=69, srv860530246=92, srv907474950=94, srv668320237=86, srv1029743639=1, srv1081755690=8, srv1520119812=35, srv1636995353=38, srv1262678352=22, srv1987047779=56, srv1625454769=37, srv53155336=82, srv1941673759=53, srv1279319560=24, srv460958600=80, srv619881801=84, srv2137399537=64, srv380077402=75, srv973117251=97, srv320435695=72, srv144785900=33, srv2114182525=62, srv1194317391=16, srv1035276344=2, srv1379481063=28, srv1946111224=54, srv1142045055=12, srv1399545454=29, srv218231011=67, srv2118112781=63, srv1094536021=9, srv1010874062=0, srv125551427=20, srv1837996624=50, srv863266028=93, srv1950504933=55, srv379973383=74, srv1990372834=57, srv685732687=87, srv1336769151=25, srv1834719377=49, srv1246219718=19, srv1641877846=39, srv595405446=83, srv1839197841=51, srv1767750751=45, srv496767162=81, srv450749582=78, srv1261025374=21, srv2083631045=60, srv916492033=95, srv2089077927=61, srv1691512634=42} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-28 22:53:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-28 22:53:20,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-28 22:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-28 22:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-28 22:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-28 22:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-28 22:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-28 22:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-28 22:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-28 22:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-28 22:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-28 22:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-28 22:53:20,125 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-28 22:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-28 22:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-28 22:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-28 22:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-28 22:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-28 22:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-28 22:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-28 22:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-28 22:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-28 22:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-28 22:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-28 22:53:20,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-28 22:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-28 22:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-28 22:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-28 22:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-28 22:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-28 22:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-28 22:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-28 22:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-28 22:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-28 22:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-28 22:53:20,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-28 22:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-28 22:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-28 22:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-28 22:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-28 22:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-28 22:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-28 22:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-28 22:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-28 22:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-28 22:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-28 22:53:20,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-28 22:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-28 22:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-28 22:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-28 22:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-28 22:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-28 22:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-28 22:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-28 22:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-28 22:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-28 22:53:20,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-28 22:53:20,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-28 22:53:20,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-28 22:53:20,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-28 22:53:20,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-28 22:53:20,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-28 22:53:20,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-28 22:53:20,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-28 22:53:20,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-28 22:53:20,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-28 22:53:20,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-28 22:53:20,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-28 22:53:20,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-28 22:53:20,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-28 22:53:20,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-28 22:53:20,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-28 22:53:20,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-28 22:53:20,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-28 22:53:20,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-28 22:53:20,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-28 22:53:20,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-28 22:53:20,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-28 22:53:20,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-28 22:53:20,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-28 22:53:20,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-28 22:53:20,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-28 22:53:20,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-28 22:53:20,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-28 22:53:20,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-28 22:53:20,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-28 22:53:20,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-28 22:53:20,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-28 22:53:20,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-28 22:53:20,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-28 22:53:20,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-28 22:53:20,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-28 22:53:20,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-28 22:53:20,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-28 22:53:20,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-28 22:53:20,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-28 22:53:20,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-28 22:53:20,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-28 22:53:20,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-28 22:53:20,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-28 22:53:20,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-28 22:53:20,271 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=9.50098893244694E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-28 22:53:20,272 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1261025374:83 , srv1365271214:83 , srv1423318811:83 , srv2036826827:83 , srv390668236:83 , srv860530246:83 , srv863266028:83 , srv916492033:83 , srv1010874062:84 , srv106508637:84 , srv1076181606:84 , srv1094536021:84 , srv1184211421:84 , srv1187345432:84 , srv1207833815:84 , srv1208866723:84 , srv1246219718:84 , srv1279319560:84 , srv1336769151:84 , srv137542090:84 , srv1379481063:84 , srv1423010703:84 , srv144785900:84 , srv1456537538:84 , srv1520119812:84 , srv1667961107:84 , srv1691512634:84 , srv175017347:84 , srv1767750751:84 , srv1789751244:84 , srv1795939254:84 , srv1837996624:84 , srv1941673759:84 , srv1987047779:84 , srv1990372834:84 , srv2083631045:84 , srv2089077927:84 , srv2137399537:84 , srv218231011:84 , srv235065032:84 , srv267487181:84 , srv320435695:84 , srv355268967:84 , srv380077402:84 , srv382751709:84 , srv450749582:84 , srv496767162:84 , srv53155336:84 , srv619881801:84 , srv661519157:84 , srv6907637:84 , srv746044696:84 , srv990894234:84 , srv1035276344:85 , srv1043992105:85 , srv105219275:85 , srv1072906082:85 , srv113028445:85 , srv1138700888:85 , srv1170163738:85 , srv1194317391:85 , srv125551427:85 , srv1262678352:85 , srv1269722399:85 , srv1399545454:85 , srv1410539506:85 , srv1557899073:85 , srv1625454769:85 , srv1636995353:85 , srv1641877846:85 , srv1669134071:85 , srv1699429575:85 , srv1768998369:85 , srv1834719377:85 , srv1839197841:85 , srv1870530017:85 , srv1946111224:85 , srv1950504933:85 , srv2039564257:85 , srv2114182525:85 , srv2118112781:85 , srv2139088525:85 , srv2142252987:85 , srv251650302:85 , srv260188711:85 , srv379973383:85 , srv452031461:85 , srv460958600:85 , srv595405446:85 , srv668320237:85 , srv685732687:85 , srv75142071:85 , srv841869434:85 , srv907474950:85 , srv930243914:85 , srv973117251:85 , srv97383712:85 , srv1081755690:86 , srv1142045055:86 , srv1029743639:644 }
2024-01-28 22:53:20,304 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=285 (was 285), ProcessCount=167 (was 167), AvailableMemoryMB=2585 (was 3388)
2024-01-28 22:53:20,315 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=285, ProcessCount=167, AvailableMemoryMB=2584
2024-01-28 22:53:20,315 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-28 22:53:20,316 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-01-28 22:53:20,332 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1052499048:84 , srv1076594846:84 , srv1089956293:84 , srv1247771353:84 , srv1304755635:84 , srv1338181817:84 , srv1355882326:84 , srv1602803124:84 , srv1669206739:84 , srv167349696:84 , srv1778053211:84 , srv1804036738:84 , srv1976409621:84 , srv1976689392:84 , srv2029281218:84 , srv2092039814:84 , srv2099856101:84 , srv367059611:84 , srv473736418:84 , srv504024585:84 , srv548726112:84 , srv732992235:84 , srv769519343:84 , srv781385759:84 , srv788795971:84 , srv805856397:84 , srv809142244:84 , srv873880671:84 , srv919230168:84 , srv545997130:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-01-28 22:53:20,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-28 22:53:20,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1976689392=13, srv1247771353=3, srv2029281218=14, srv732992235=22, srv1052499048=0, srv2099856101=16, srv809142244=27, srv504024585=19, srv1778053211=10, srv1976409621=12, srv1304755635=4, srv805856397=26, srv367059611=17, srv1089956293=2, srv473736418=18, srv919230168=29, srv548726112=21, srv545997130=20, srv1338181817=5, srv2092039814=15, srv788795971=25, srv1355882326=6, srv769519343=23, srv1602803124=7, srv873880671=28, srv1669206739=8, srv167349696=9, srv1076594846=1, srv1804036738=11, srv781385759=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-28 22:53:20,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-28 22:53:20,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-28 22:53:20,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-28 22:53:20,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-28 22:53:20,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-28 22:53:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-28 22:53:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-28 22:53:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-28 22:53:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-28 22:53:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-28 22:53:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-28 22:53:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-28 22:53:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-28 22:53:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-28 22:53:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-28 22:53:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-28 22:53:20,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-28 22:53:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-28 22:53:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-28 22:53:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-28 22:53:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-28 22:53:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-28 22:53:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-28 22:53:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-28 22:53:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-28 22:53:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-28 22:53:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-28 22:53:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-28 22:53:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-28 22:53:20,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-28 22:53:20,351 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-01-28 22:53:20,375 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0912842573812227, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-28 22:53:20,376 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912480676978583, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0912842573812227, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-01-28 22:55:20,355 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6997174 different iterations.  Found a solution that moves 1856 regions; Going from a computed imbalance of 0.9912480676978583 to a new imbalance of 4.3319965896196796E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6874074074074074, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-28 22:55:20,362 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1052499048:90 , srv1076594846:90 , srv1089956293:90 , srv1247771353:90 , srv1304755635:90 , srv1338181817:90 , srv1355882326:90 , srv1602803124:90 , srv1669206739:90 , srv167349696:90 , srv1778053211:90 , srv1804036738:90 , srv1976409621:90 , srv1976689392:90 , srv2029281218:90 , srv2092039814:90 , srv2099856101:90 , srv367059611:90 , srv473736418:90 , srv504024585:90 , srv545997130:90 , srv548726112:90 , srv732992235:90 , srv769519343:90 , srv781385759:90 , srv788795971:90 , srv805856397:90 , srv809142244:90 , srv873880671:90 , srv919230168:90 }
2024-01-28 22:55:20,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-28 22:55:20,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1976689392=13, srv1247771353=3, srv2029281218=14, srv732992235=22, srv1052499048=0, srv2099856101=16, srv809142244=27, srv504024585=19, srv1778053211=10, srv1976409621=12, srv1304755635=4, srv805856397=26, srv367059611=17, srv1089956293=2, srv473736418=18, srv919230168=29, srv548726112=21, srv545997130=20, srv1338181817=5, srv2092039814=15, srv788795971=25, srv1355882326=6, srv769519343=23, srv1602803124=7, srv873880671=28, srv1669206739=8, srv167349696=9, srv1076594846=1, srv1804036738=11, srv781385759=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-28 22:55:20,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-28 22:55:20,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-28 22:55:20,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-28 22:55:20,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-28 22:55:20,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-28 22:55:20,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-28 22:55:20,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-28 22:55:20,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-28 22:55:20,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-28 22:55:20,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-28 22:55:20,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-28 22:55:20,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-28 22:55:20,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-28 22:55:20,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-28 22:55:20,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-28 22:55:20,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-28 22:55:20,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-28 22:55:20,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-28 22:55:20,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-28 22:55:20,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-28 22:55:20,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-28 22:55:20,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-28 22:55:20,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-28 22:55:20,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-28 22:55:20,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-28 22:55:20,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-28 22:55:20,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-28 22:55:20,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-28 22:55:20,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-28 22:55:20,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-28 22:55:20,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-01-28 22:55:20,381 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-28 22:55:20,381 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1052499048:90 , srv1076594846:90 , srv1089956293:90 , srv1247771353:90 , srv1304755635:90 , srv1338181817:90 , srv1355882326:90 , srv1602803124:90 , srv1669206739:90 , srv167349696:90 , srv1778053211:90 , srv1804036738:90 , srv1976409621:90 , srv1976689392:90 , srv2029281218:90 , srv2092039814:90 , srv2099856101:90 , srv367059611:90 , srv473736418:90 , srv504024585:90 , srv545997130:90 , srv548726112:90 , srv732992235:90 , srv769519343:90 , srv781385759:90 , srv788795971:90 , srv805856397:90 , srv809142244:90 , srv873880671:90 , srv919230168:90 }
2024-01-28 22:55:20,393 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=239 (was 285), ProcessCount=167 (was 167), AvailableMemoryMB=2805 (was 2584) - AvailableMemoryMB LEAK? -