2023-08-09 04:53:10,132 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-08-09 04:53:10,469 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-08-09 04:53:10,592 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-08-09 04:53:10,661 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-08-09 04:53:10,671 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-08-09 04:53:10,687 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=246, ProcessCount=174, AvailableMemoryMB=2529
2023-08-09 04:53:10,689 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-08-09 04:53:10,689 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-08-09 04:53:10,842 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1018623635:84 , srv1059973772:84 , srv106769051:84 , srv112156127:84 , srv1129698475:84 , srv114800282:84 , srv1151005220:84 , srv1193665016:84 , srv1265594066:84 , srv1287033245:84 , srv1293510105:84 , srv1342618209:84 , srv135721240:84 , srv1358852821:84 , srv1386995675:84 , srv1412206937:84 , srv1417968277:84 , srv1427597009:84 , srv1432817660:84 , srv1441095568:84 , srv1453720848:84 , srv1477772591:84 , srv1496887042:84 , srv151650185:84 , srv1540643053:84 , srv1546731300:84 , srv1556828771:84 , srv1562422399:84 , srv1578833847:84 , srv158038689:84 , srv1586952929:84 , srv16224992:84 , srv1630578377:84 , srv1646687460:84 , srv1684555663:84 , srv1685549444:84 , srv1689816081:84 , srv1697014852:84 , srv1710417897:84 , srv1773045589:84 , srv1784061022:84 , srv1790653471:84 , srv1823196916:84 , srv1832023397:84 , srv1885132761:84 , srv1895355361:84 , srv1903300044:84 , srv1904218294:84 , srv1905572029:84 , srv1921948446:84 , srv199422394:84 , srv2002481733:84 , srv2050380710:84 , srv2131287921:84 , srv213674544:84 , srv214041005:84 , srv259299134:84 , srv267942587:84 , srv286971509:84 , srv321182971:84 , srv32627012:84 , srv358635453:84 , srv358810055:84 , srv377189173:84 , srv377902388:84 , srv396533439:84 , srv404206040:84 , srv409657778:84 , srv467377169:84 , srv472328205:84 , srv475972221:84 , srv489047925:84 , srv496163075:84 , srv514951700:84 , srv531785811:84 , srv545116235:84 , srv563058470:84 , srv580692096:84 , srv629975555:84 , srv632769746:84 , srv658580786:84 , srv658865334:84 , srv720165865:84 , srv727016602:84 , srv740310720:84 , srv744023651:84 , srv74482232:84 , srv758361957:84 , srv775709120:84 , srv805971259:84 , srv823179551:84 , srv830633234:84 , srv857731699:84 , srv874037811:84 , srv908950289:84 , srv917172563:84 , srv960004037:84 , srv967654647:84 , srv976266330:84 , srv352160384:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-08-09 04:53:10,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-09 04:53:10,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1832023397=43, srv1496887042=22, srv1710417897=38, srv632769746=80, srv514951700=74, srv1562422399=27, srv358635453=62, srv1265594066=8, srv629975555=79, srv1412206937=15, srv158038689=29, srv467377169=69, srv2002481733=51, srv917172563=96, srv16224992=31, srv720165865=83, srv267942587=57, srv1823196916=42, srv960004037=97, srv967654647=98, srv74482232=87, srv32627012=60, srv580692096=78, srv1018623635=0, srv199422394=50, srv1441095568=19, srv1684555663=34, srv1432817660=18, srv1578833847=28, srv1129698475=4, srv1151005220=6, srv377189173=64, srv1386995675=14, srv1427597009=17, srv1905572029=48, srv1293510105=10, srv908950289=95, srv135721240=12, srv404206040=67, srv545116235=76, srv857731699=93, srv377902388=65, srv658580786=81, srv409657778=68, srv830633234=92, srv1546731300=25, srv1193665016=7, srv658865334=82, srv758361957=88, srv1586952929=30, srv2131287921=53, srv1556828771=26, srv1904218294=47, srv563058470=77, srv1921948446=49, srv1689816081=36, srv472328205=70, srv823179551=91, srv2050380710=52, srv1477772591=21, srv112156127=3, srv740310720=85, srv744023651=86, srv1773045589=39, srv1358852821=13, srv1885132761=44, srv213674544=54, srv1685549444=35, srv1342618209=11, srv727016602=84, srv114800282=5, srv1697014852=37, srv259299134=56, srv396533439=66, srv805971259=90, srv352160384=61, srv106769051=2, srv1417968277=16, srv475972221=71, srv1540643053=24, srv489047925=72, srv976266330=99, srv1630578377=32, srv321182971=59, srv1453720848=20, srv151650185=23, srv1784061022=40, srv1059973772=1, srv496163075=73, srv1287033245=9, srv358810055=63, srv874037811=94, srv214041005=55, srv1895355361=45, srv1646687460=33, srv1790653471=41, srv286971509=58, srv1903300044=46, srv775709120=89, srv531785811=75} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-09 04:53:10,948 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-09 04:53:10,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-09 04:53:10,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-09 04:53:10,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-09 04:53:10,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-09 04:53:10,949 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-09 04:53:10,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-09 04:53:10,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-09 04:53:10,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-09 04:53:10,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-09 04:53:10,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-09 04:53:10,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-09 04:53:10,950 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-09 04:53:10,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-09 04:53:10,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-09 04:53:10,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-09 04:53:10,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-09 04:53:10,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-09 04:53:10,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-09 04:53:10,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-09 04:53:10,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-09 04:53:10,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-09 04:53:10,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-09 04:53:10,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-09 04:53:10,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-09 04:53:10,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-09 04:53:10,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-09 04:53:10,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-09 04:53:10,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-09 04:53:10,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-09 04:53:10,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-09 04:53:10,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-09 04:53:10,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-09 04:53:10,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-09 04:53:10,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-09 04:53:10,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-09 04:53:10,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-09 04:53:10,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-09 04:53:10,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-09 04:53:10,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-09 04:53:10,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-09 04:53:10,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-09 04:53:10,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-09 04:53:10,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-09 04:53:10,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-09 04:53:10,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-09 04:53:10,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-09 04:53:10,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-09 04:53:10,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-09 04:53:10,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-09 04:53:10,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-09 04:53:10,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-09 04:53:10,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-09 04:53:10,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-09 04:53:10,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-09 04:53:10,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-09 04:53:10,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-09 04:53:10,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-09 04:53:10,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-09 04:53:10,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-09 04:53:10,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-09 04:53:10,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-09 04:53:10,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-09 04:53:10,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-09 04:53:10,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-09 04:53:10,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-09 04:53:10,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-09 04:53:10,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-09 04:53:10,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-09 04:53:10,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-09 04:53:10,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-09 04:53:10,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-09 04:53:10,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-09 04:53:10,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-09 04:53:10,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-09 04:53:10,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-09 04:53:10,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-09 04:53:10,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-09 04:53:10,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-09 04:53:10,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-09 04:53:10,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-09 04:53:10,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-09 04:53:10,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-09 04:53:10,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-09 04:53:10,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-09 04:53:10,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-09 04:53:10,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-09 04:53:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-09 04:53:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-09 04:53:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-09 04:53:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-09 04:53:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-09 04:53:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-09 04:53:10,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-09 04:53:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-09 04:53:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-09 04:53:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-09 04:53:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-09 04:53:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-09 04:53:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-09 04:53:10,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-09 04:53:11,366 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-09 04:53:11,367 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-08-09 04:53:11,367 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-08-09 04:53:12,241 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1249 ms to try 5000 different iterations.  Found a solution that moves 4685 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17880292714780052. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03995539350960699, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05507876209275253, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5205555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03995539350960699, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17383333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-09 04:53:12,284 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv830633234:79 , srv199422394:80 , srv917172563:80 , srv1562422399:81 , srv1293510105:82 , srv1685549444:82 , srv1018623635:83 , srv1823196916:83 , srv214041005:83 , srv727016602:83 , srv740310720:83 , srv1193665016:84 , srv135721240:84 , srv1441095568:84 , srv158038689:84 , srv1586952929:84 , srv1697014852:84 , srv1710417897:84 , srv1905572029:84 , srv2002481733:84 , srv213674544:84 , srv531785811:84 , srv632769746:84 , srv823179551:84 , srv1151005220:85 , srv1453720848:85 , srv1684555663:85 , srv1784061022:85 , srv1832023397:85 , srv1903300044:85 , srv259299134:85 , srv321182971:85 , srv467377169:85 , srv496163075:85 , srv514951700:85 , srv967654647:85 , srv1129698475:86 , srv1265594066:86 , srv1287033245:86 , srv1342618209:86 , srv1358852821:86 , srv1432817660:86 , srv1496887042:86 , srv151650185:86 , srv1578833847:86 , srv1646687460:86 , srv1904218294:86 , srv2050380710:86 , srv2131287921:86 , srv267942587:86 , srv377902388:86 , srv396533439:86 , srv857731699:86 , srv112156127:87 , srv1386995675:87 , srv1417968277:87 , srv1540643053:87 , srv1895355361:87 , srv377189173:87 , srv475972221:87 , srv545116235:87 , srv744023651:87 , srv1059973772:88 , srv1412206937:88 , srv1556828771:88 , srv1630578377:88 , srv1790653471:88 , srv32627012:88 , srv358810055:88 , srv404206040:88 , srv472328205:88 , srv489047925:88 , srv580692096:88 , srv720165865:88 , srv74482232:88 , srv775709120:88 , srv874037811:88 , srv114800282:89 , srv1689816081:89 , srv1921948446:89 , srv286971509:89 , srv358635453:89 , srv629975555:89 , srv658580786:89 , srv908950289:89 , srv976266330:89 , srv106769051:90 , srv1427597009:90 , srv1477772591:90 , srv16224992:90 , srv1773045589:90 , srv409657778:90 , srv658865334:90 , srv758361957:90 , srv805971259:90 , srv1885132761:91 , srv563058470:91 , srv960004037:91 , srv1546731300:95 , srv352160384:445 }
2023-08-09 04:53:12,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-09 04:53:12,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1832023397=43, srv1496887042=22, srv1710417897=38, srv632769746=80, srv514951700=74, srv1562422399=27, srv358635453=62, srv1265594066=8, srv629975555=79, srv1412206937=15, srv158038689=29, srv467377169=69, srv2002481733=51, srv917172563=96, srv16224992=31, srv720165865=83, srv267942587=57, srv1823196916=42, srv960004037=97, srv967654647=98, srv74482232=87, srv32627012=60, srv580692096=78, srv1018623635=0, srv199422394=50, srv1441095568=19, srv1684555663=34, srv1432817660=18, srv1578833847=28, srv1129698475=4, srv1151005220=6, srv377189173=64, srv1386995675=14, srv1427597009=17, srv1905572029=48, srv1293510105=10, srv908950289=95, srv135721240=12, srv404206040=67, srv545116235=76, srv857731699=93, srv377902388=65, srv658580786=81, srv409657778=68, srv830633234=92, srv1546731300=25, srv1193665016=7, srv658865334=82, srv758361957=88, srv1586952929=30, srv2131287921=53, srv1556828771=26, srv1904218294=47, srv563058470=77, srv1921948446=49, srv1689816081=36, srv472328205=70, srv823179551=91, srv2050380710=52, srv1477772591=21, srv112156127=3, srv740310720=85, srv744023651=86, srv1773045589=39, srv1358852821=13, srv1885132761=44, srv213674544=54, srv1685549444=35, srv1342618209=11, srv727016602=84, srv114800282=5, srv1697014852=37, srv259299134=56, srv396533439=66, srv805971259=90, srv352160384=61, srv106769051=2, srv1417968277=16, srv475972221=71, srv1540643053=24, srv489047925=72, srv976266330=99, srv1630578377=32, srv321182971=59, srv1453720848=20, srv151650185=23, srv1784061022=40, srv1059973772=1, srv496163075=73, srv1287033245=9, srv358810055=63, srv874037811=94, srv214041005=55, srv1895355361=45, srv1646687460=33, srv1790653471=41, srv286971509=58, srv1903300044=46, srv775709120=89, srv531785811=75} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-09 04:53:12,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-09 04:53:12,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-09 04:53:12,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-09 04:53:12,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-09 04:53:12,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-09 04:53:12,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-09 04:53:12,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-09 04:53:12,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-09 04:53:12,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-09 04:53:12,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-09 04:53:12,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-09 04:53:12,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-09 04:53:12,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-09 04:53:12,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-09 04:53:12,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-09 04:53:12,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-09 04:53:12,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-09 04:53:12,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-09 04:53:12,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-09 04:53:12,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-09 04:53:12,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-09 04:53:12,319 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-09 04:53:12,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-09 04:53:12,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-09 04:53:12,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-09 04:53:12,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-09 04:53:12,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-09 04:53:12,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-09 04:53:12,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-09 04:53:12,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-09 04:53:12,320 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-09 04:53:12,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-09 04:53:12,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-09 04:53:12,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-09 04:53:12,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-09 04:53:12,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-09 04:53:12,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-09 04:53:12,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-09 04:53:12,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-09 04:53:12,321 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-09 04:53:12,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-09 04:53:12,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-09 04:53:12,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-09 04:53:12,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-09 04:53:12,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-09 04:53:12,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-09 04:53:12,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-09 04:53:12,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-09 04:53:12,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-09 04:53:12,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-09 04:53:12,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-09 04:53:12,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-09 04:53:12,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-09 04:53:12,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-09 04:53:12,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-09 04:53:12,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-09 04:53:12,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-09 04:53:12,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-09 04:53:12,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-09 04:53:12,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-09 04:53:12,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-09 04:53:12,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-09 04:53:12,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-09 04:53:12,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-09 04:53:12,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-09 04:53:12,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-09 04:53:12,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-09 04:53:12,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-09 04:53:12,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-09 04:53:12,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-09 04:53:12,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-09 04:53:12,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-09 04:53:12,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-09 04:53:12,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-09 04:53:12,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-09 04:53:12,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-09 04:53:12,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-09 04:53:12,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-09 04:53:12,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-09 04:53:12,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-09 04:53:12,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-09 04:53:12,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-09 04:53:12,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-09 04:53:12,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-09 04:53:12,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-09 04:53:12,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-09 04:53:12,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-09 04:53:12,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-09 04:53:12,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-09 04:53:12,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-09 04:53:12,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-09 04:53:12,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-09 04:53:12,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-09 04:53:12,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-09 04:53:12,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-09 04:53:12,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-09 04:53:12,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-09 04:53:12,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-09 04:53:12,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-09 04:53:12,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-09 04:53:12,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-09 04:53:12,560 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03995539350960699, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05507876209275253, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03995539350960699, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17383333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-09 04:53:12,561 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-08-09 04:53:12,561 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1787701220766437, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03995539350960699, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05507876209275253, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03995539350960699, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17383333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-08-09 04:53:12,863 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 511 ms to try 5000 different iterations.  Found a solution that moves 2658 regions; Going from a computed imbalance of 0.1787701220766437 to a new imbalance of 0.015319866419087686. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02060683082119442); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03218562882066108, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29533333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02060683082119442); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04475, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-09 04:53:12,875 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1546731300:76 , srv16224992:78 , srv1477772591:79 , srv563058470:80 , srv805971259:80 , srv960004037:80 , srv106769051:81 , srv1059973772:82 , srv114800282:82 , srv1885132761:82 , srv976266330:82 , srv1556828771:83 , srv1578833847:83 , srv1689816081:83 , srv1773045589:83 , srv1921948446:83 , srv286971509:83 , srv358635453:83 , srv472328205:83 , srv475972221:83 , srv629975555:83 , srv658580786:83 , srv658865334:83 , srv720165865:83 , srv775709120:83 , srv1342618209:84 , srv1427597009:84 , srv1496887042:84 , srv1630578377:84 , srv404206040:84 , srv409657778:84 , srv580692096:84 , srv74482232:84 , srv758361957:84 , srv112156127:85 , srv1129698475:85 , srv1287033245:85 , srv1417968277:85 , srv321182971:85 , srv358810055:85 , srv377189173:85 , srv377902388:85 , srv396533439:85 , srv874037811:85 , srv908950289:85 , srv1151005220:86 , srv1358852821:86 , srv1386995675:86 , srv1432817660:86 , srv1540643053:86 , srv1697014852:86 , srv1784061022:86 , srv1895355361:86 , srv2131287921:86 , srv213674544:86 , srv267942587:86 , srv489047925:86 , srv514951700:86 , srv1265594066:87 , srv1441095568:87 , srv1453720848:87 , srv1832023397:87 , srv1904218294:87 , srv32627012:87 , srv545116235:87 , srv857731699:87 , srv967654647:87 , srv135721240:88 , srv1412206937:88 , srv151650185:88 , srv1646687460:88 , srv1684555663:88 , srv1790653471:88 , srv1823196916:88 , srv1905572029:88 , srv2002481733:88 , srv214041005:88 , srv467377169:88 , srv496163075:88 , srv531785811:88 , srv727016602:88 , srv823179551:88 , srv1293510105:89 , srv158038689:89 , srv1710417897:89 , srv1903300044:89 , srv2050380710:89 , srv259299134:89 , srv632769746:89 , srv740310720:89 , srv744023651:89 , srv1018623635:90 , srv1193665016:90 , srv1586952929:90 , srv1685549444:91 , srv917172563:91 , srv199422394:92 , srv1562422399:93 , srv830633234:94 , srv352160384:512 }
2023-08-09 04:53:12,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-09 04:53:12,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1832023397=43, srv1496887042=22, srv1710417897=38, srv632769746=80, srv514951700=74, srv1562422399=27, srv358635453=62, srv1265594066=8, srv629975555=79, srv1412206937=15, srv158038689=29, srv467377169=69, srv2002481733=51, srv917172563=96, srv16224992=31, srv720165865=83, srv267942587=57, srv1823196916=42, srv960004037=97, srv967654647=98, srv74482232=87, srv32627012=60, srv580692096=78, srv1018623635=0, srv199422394=50, srv1441095568=19, srv1684555663=34, srv1432817660=18, srv1578833847=28, srv1129698475=4, srv1151005220=6, srv377189173=64, srv1386995675=14, srv1427597009=17, srv1905572029=48, srv1293510105=10, srv908950289=95, srv135721240=12, srv404206040=67, srv545116235=76, srv857731699=93, srv377902388=65, srv658580786=81, srv409657778=68, srv830633234=92, srv1546731300=25, srv1193665016=7, srv658865334=82, srv758361957=88, srv1586952929=30, srv2131287921=53, srv1556828771=26, srv1904218294=47, srv563058470=77, srv1921948446=49, srv1689816081=36, srv472328205=70, srv823179551=91, srv2050380710=52, srv1477772591=21, srv112156127=3, srv740310720=85, srv744023651=86, srv1773045589=39, srv1358852821=13, srv1885132761=44, srv213674544=54, srv1685549444=35, srv1342618209=11, srv727016602=84, srv114800282=5, srv1697014852=37, srv259299134=56, srv396533439=66, srv805971259=90, srv352160384=61, srv106769051=2, srv1417968277=16, srv475972221=71, srv1540643053=24, srv489047925=72, srv976266330=99, srv1630578377=32, srv321182971=59, srv1453720848=20, srv151650185=23, srv1784061022=40, srv1059973772=1, srv496163075=73, srv1287033245=9, srv358810055=63, srv874037811=94, srv214041005=55, srv1895355361=45, srv1646687460=33, srv1790653471=41, srv286971509=58, srv1903300044=46, srv775709120=89, srv531785811=75} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-09 04:53:12,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-09 04:53:12,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-09 04:53:12,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-09 04:53:12,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-09 04:53:12,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-09 04:53:12,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-09 04:53:12,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-09 04:53:12,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-09 04:53:12,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-09 04:53:12,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-09 04:53:12,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-09 04:53:12,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-09 04:53:12,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-09 04:53:12,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-09 04:53:12,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-09 04:53:12,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-09 04:53:12,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-09 04:53:12,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-09 04:53:12,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-09 04:53:12,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-09 04:53:12,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-09 04:53:12,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-09 04:53:12,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-09 04:53:12,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-09 04:53:12,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-09 04:53:12,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-09 04:53:12,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-09 04:53:12,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-09 04:53:12,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-09 04:53:12,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-09 04:53:12,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-09 04:53:12,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-09 04:53:12,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-09 04:53:12,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-09 04:53:12,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-09 04:53:12,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-09 04:53:12,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-09 04:53:12,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-09 04:53:12,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-09 04:53:12,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-09 04:53:12,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-09 04:53:12,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-09 04:53:12,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-09 04:53:12,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-09 04:53:12,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-09 04:53:12,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-09 04:53:12,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-09 04:53:12,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-09 04:53:12,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-09 04:53:12,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-09 04:53:12,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-09 04:53:12,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-09 04:53:12,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-09 04:53:12,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-09 04:53:12,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-09 04:53:12,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-09 04:53:12,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-09 04:53:12,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-09 04:53:12,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-09 04:53:12,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-09 04:53:12,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-09 04:53:12,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-09 04:53:12,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-09 04:53:12,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-09 04:53:12,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-09 04:53:12,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-09 04:53:12,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-09 04:53:12,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-09 04:53:12,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-09 04:53:12,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-09 04:53:12,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-09 04:53:12,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-09 04:53:12,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-09 04:53:12,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-09 04:53:12,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-09 04:53:12,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-09 04:53:12,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-09 04:53:12,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-09 04:53:12,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-09 04:53:12,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-09 04:53:12,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-09 04:53:12,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-09 04:53:12,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-09 04:53:12,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-09 04:53:12,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-09 04:53:12,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-09 04:53:12,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-09 04:53:12,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-09 04:53:12,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-09 04:53:12,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-09 04:53:12,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-09 04:53:12,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-09 04:53:12,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-09 04:53:12,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-09 04:53:12,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-09 04:53:12,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-09 04:53:12,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-09 04:53:12,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-09 04:53:12,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-09 04:53:12,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-09 04:53:12,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-09 04:53:13,072 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02060683082119442); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03218562882066108, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02060683082119442); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04475, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-09 04:53:13,073 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-08-09 04:53:13,073 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015301254705291551, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02060683082119442); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03218562882066108, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02060683082119442); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01225); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04475, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-08-09 04:53:13,391 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 441 ms to try 5000 different iterations.  Found a solution that moves 924 regions; Going from a computed imbalance of 0.015301254705291551 to a new imbalance of 5.889422139719197E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005877325962449793); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008109796489563252); MoveCostFunction : (multiplier=7.0, imbalance=0.10266666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005877325962449793); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-09 04:53:13,394 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1412206937:82 , srv1790653471:82 , srv32627012:82 , srv744023651:82 , srv1586952929:83 , srv1646687460:83 , srv1685549444:83 , srv1903300044:83 , srv2050380710:83 , srv409657778:83 , srv545116235:83 , srv758361957:83 , srv857731699:83 , srv1018623635:84 , srv1193665016:84 , srv1417968277:84 , srv1427597009:84 , srv151650185:84 , srv1540643053:84 , srv1562422399:84 , srv158038689:84 , srv1684555663:84 , srv1710417897:84 , srv1885132761:84 , srv467377169:84 , srv489047925:84 , srv658865334:84 , srv830633234:84 , srv874037811:84 , srv908950289:84 , srv112156127:85 , srv1265594066:85 , srv135721240:85 , srv1358852821:85 , srv1386995675:85 , srv1432817660:85 , srv1453720848:85 , srv1630578377:85 , srv1773045589:85 , srv1895355361:85 , srv1904218294:85 , srv1905572029:85 , srv1921948446:85 , srv2131287921:85 , srv259299134:85 , srv358635453:85 , srv358810055:85 , srv496163075:85 , srv580692096:85 , srv632769746:85 , srv658580786:85 , srv74482232:85 , srv106769051:86 , srv114800282:86 , srv1689816081:86 , srv1823196916:86 , srv1832023397:86 , srv199422394:86 , srv2002481733:86 , srv267942587:86 , srv286971509:86 , srv377189173:86 , srv472328205:86 , srv514951700:86 , srv531785811:86 , srv563058470:86 , srv720165865:86 , srv727016602:86 , srv740310720:86 , srv775709120:86 , srv823179551:86 , srv917172563:86 , srv960004037:86 , srv967654647:86 , srv976266330:86 , srv1129698475:87 , srv1151005220:87 , srv1287033245:87 , srv1293510105:87 , srv1441095568:87 , srv1556828771:87 , srv1697014852:87 , srv1784061022:87 , srv213674544:87 , srv214041005:87 , srv321182971:87 , srv377902388:87 , srv396533439:87 , srv404206040:87 , srv475972221:87 , srv629975555:87 , srv805971259:87 , srv1059973772:88 , srv1342618209:88 , srv1477772591:88 , srv1496887042:88 , srv1546731300:88 , srv1578833847:88 , srv16224992:89 , srv352160384:553 }
2023-08-09 04:53:13,401 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-09 04:53:13,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1832023397=43, srv1496887042=22, srv1710417897=38, srv632769746=80, srv514951700=74, srv1562422399=27, srv358635453=62, srv1265594066=8, srv629975555=79, srv1412206937=15, srv158038689=29, srv467377169=69, srv2002481733=51, srv917172563=96, srv16224992=31, srv720165865=83, srv267942587=57, srv1823196916=42, srv960004037=97, srv967654647=98, srv74482232=87, srv32627012=60, srv580692096=78, srv1018623635=0, srv199422394=50, srv1441095568=19, srv1684555663=34, srv1432817660=18, srv1578833847=28, srv1129698475=4, srv1151005220=6, srv377189173=64, srv1386995675=14, srv1427597009=17, srv1905572029=48, srv1293510105=10, srv908950289=95, srv135721240=12, srv404206040=67, srv545116235=76, srv857731699=93, srv377902388=65, srv658580786=81, srv409657778=68, srv830633234=92, srv1546731300=25, srv1193665016=7, srv658865334=82, srv758361957=88, srv1586952929=30, srv2131287921=53, srv1556828771=26, srv1904218294=47, srv563058470=77, srv1921948446=49, srv1689816081=36, srv472328205=70, srv823179551=91, srv2050380710=52, srv1477772591=21, srv112156127=3, srv740310720=85, srv744023651=86, srv1773045589=39, srv1358852821=13, srv1885132761=44, srv213674544=54, srv1685549444=35, srv1342618209=11, srv727016602=84, srv114800282=5, srv1697014852=37, srv259299134=56, srv396533439=66, srv805971259=90, srv352160384=61, srv106769051=2, srv1417968277=16, srv475972221=71, srv1540643053=24, srv489047925=72, srv976266330=99, srv1630578377=32, srv321182971=59, srv1453720848=20, srv151650185=23, srv1784061022=40, srv1059973772=1, srv496163075=73, srv1287033245=9, srv358810055=63, srv874037811=94, srv214041005=55, srv1895355361=45, srv1646687460=33, srv1790653471=41, srv286971509=58, srv1903300044=46, srv775709120=89, srv531785811=75} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-09 04:53:13,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-09 04:53:13,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-09 04:53:13,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-09 04:53:13,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-09 04:53:13,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-09 04:53:13,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-09 04:53:13,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-09 04:53:13,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-09 04:53:13,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-09 04:53:13,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-09 04:53:13,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-09 04:53:13,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-09 04:53:13,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-09 04:53:13,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-09 04:53:13,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-09 04:53:13,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-09 04:53:13,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-09 04:53:13,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-09 04:53:13,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-09 04:53:13,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-09 04:53:13,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-09 04:53:13,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-09 04:53:13,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-09 04:53:13,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-09 04:53:13,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-09 04:53:13,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-09 04:53:13,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-09 04:53:13,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-09 04:53:13,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-09 04:53:13,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-09 04:53:13,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-09 04:53:13,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-09 04:53:13,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-09 04:53:13,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-09 04:53:13,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-09 04:53:13,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-09 04:53:13,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-09 04:53:13,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-09 04:53:13,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-09 04:53:13,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-09 04:53:13,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-09 04:53:13,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-09 04:53:13,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-09 04:53:13,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-09 04:53:13,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-09 04:53:13,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-09 04:53:13,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-09 04:53:13,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-09 04:53:13,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-09 04:53:13,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-09 04:53:13,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-09 04:53:13,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-09 04:53:13,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-09 04:53:13,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-09 04:53:13,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-09 04:53:13,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-09 04:53:13,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-09 04:53:13,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-09 04:53:13,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-09 04:53:13,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-09 04:53:13,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-09 04:53:13,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-09 04:53:13,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-09 04:53:13,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-09 04:53:13,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-09 04:53:13,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-09 04:53:13,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-09 04:53:13,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-09 04:53:13,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-09 04:53:13,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-09 04:53:13,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-09 04:53:13,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-09 04:53:13,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-09 04:53:13,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-09 04:53:13,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-09 04:53:13,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-09 04:53:13,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-09 04:53:13,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-09 04:53:13,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-09 04:53:13,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-09 04:53:13,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-09 04:53:13,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-09 04:53:13,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-09 04:53:13,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-09 04:53:13,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-09 04:53:13,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-09 04:53:13,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-09 04:53:13,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-09 04:53:13,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-09 04:53:13,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-09 04:53:13,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-09 04:53:13,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-09 04:53:13,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-09 04:53:13,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-09 04:53:13,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-09 04:53:13,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-09 04:53:13,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-09 04:53:13,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-09 04:53:13,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-09 04:53:13,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-09 04:53:13,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-09 04:53:13,578 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005877325962449793); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008109796489563252); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005877325962449793); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-09 04:53:13,578 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-08-09 04:53:13,578 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.8247222768613E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005877325962449793); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008109796489563252); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005877325962449793); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-08-09 04:53:14,010 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 565 ms to try 5000 different iterations.  Found a solution that moves 266 regions; Going from a computed imbalance of 5.8247222768613E-4 to a new imbalance of 2.1295899520956506E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.029555555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-09 04:53:14,011 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1546731300:82 , srv1895355361:83 , srv259299134:83 , srv496163075:83 , srv629975555:83 , srv740310720:83 , srv1059973772:84 , srv1129698475:84 , srv114800282:84 , srv1151005220:84 , srv1193665016:84 , srv1265594066:84 , srv1287033245:84 , srv1293510105:84 , srv1342618209:84 , srv1386995675:84 , srv1412206937:84 , srv1427597009:84 , srv1441095568:84 , srv1562422399:84 , srv158038689:84 , srv1689816081:84 , srv1784061022:84 , srv1790653471:84 , srv1823196916:84 , srv1832023397:84 , srv1904218294:84 , srv199422394:84 , srv2002481733:84 , srv2050380710:84 , srv214041005:84 , srv267942587:84 , srv286971509:84 , srv377189173:84 , srv377902388:84 , srv396533439:84 , srv404206040:84 , srv467377169:84 , srv531785811:84 , srv563058470:84 , srv632769746:84 , srv744023651:84 , srv775709120:84 , srv960004037:84 , srv967654647:84 , srv1018623635:85 , srv106769051:85 , srv112156127:85 , srv135721240:85 , srv1358852821:85 , srv1432817660:85 , srv1453720848:85 , srv1477772591:85 , srv1496887042:85 , srv151650185:85 , srv1540643053:85 , srv1556828771:85 , srv1586952929:85 , srv16224992:85 , srv1630578377:85 , srv1646687460:85 , srv1684555663:85 , srv1697014852:85 , srv1710417897:85 , srv1773045589:85 , srv1885132761:85 , srv1903300044:85 , srv1905572029:85 , srv1921948446:85 , srv2131287921:85 , srv213674544:85 , srv321182971:85 , srv32627012:85 , srv358635453:85 , srv358810055:85 , srv472328205:85 , srv489047925:85 , srv514951700:85 , srv545116235:85 , srv580692096:85 , srv658580786:85 , srv658865334:85 , srv720165865:85 , srv727016602:85 , srv74482232:85 , srv758361957:85 , srv805971259:85 , srv823179551:85 , srv830633234:85 , srv857731699:85 , srv874037811:85 , srv908950289:85 , srv917172563:85 , srv976266330:85 , srv1417968277:86 , srv1578833847:86 , srv1685549444:86 , srv409657778:86 , srv475972221:86 , srv352160384:632 }
2023-08-09 04:53:14,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-09 04:53:14,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1832023397=43, srv1496887042=22, srv1710417897=38, srv632769746=80, srv514951700=74, srv1562422399=27, srv358635453=62, srv1265594066=8, srv629975555=79, srv1412206937=15, srv158038689=29, srv467377169=69, srv2002481733=51, srv917172563=96, srv16224992=31, srv720165865=83, srv267942587=57, srv1823196916=42, srv960004037=97, srv967654647=98, srv74482232=87, srv32627012=60, srv580692096=78, srv1018623635=0, srv199422394=50, srv1441095568=19, srv1684555663=34, srv1432817660=18, srv1578833847=28, srv1129698475=4, srv1151005220=6, srv377189173=64, srv1386995675=14, srv1427597009=17, srv1905572029=48, srv1293510105=10, srv908950289=95, srv135721240=12, srv404206040=67, srv545116235=76, srv857731699=93, srv377902388=65, srv658580786=81, srv409657778=68, srv830633234=92, srv1546731300=25, srv1193665016=7, srv658865334=82, srv758361957=88, srv1586952929=30, srv2131287921=53, srv1556828771=26, srv1904218294=47, srv563058470=77, srv1921948446=49, srv1689816081=36, srv472328205=70, srv823179551=91, srv2050380710=52, srv1477772591=21, srv112156127=3, srv740310720=85, srv744023651=86, srv1773045589=39, srv1358852821=13, srv1885132761=44, srv213674544=54, srv1685549444=35, srv1342618209=11, srv727016602=84, srv114800282=5, srv1697014852=37, srv259299134=56, srv396533439=66, srv805971259=90, srv352160384=61, srv106769051=2, srv1417968277=16, srv475972221=71, srv1540643053=24, srv489047925=72, srv976266330=99, srv1630578377=32, srv321182971=59, srv1453720848=20, srv151650185=23, srv1784061022=40, srv1059973772=1, srv496163075=73, srv1287033245=9, srv358810055=63, srv874037811=94, srv214041005=55, srv1895355361=45, srv1646687460=33, srv1790653471=41, srv286971509=58, srv1903300044=46, srv775709120=89, srv531785811=75} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-09 04:53:14,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-09 04:53:14,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-09 04:53:14,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-09 04:53:14,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-09 04:53:14,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-09 04:53:14,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-09 04:53:14,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-09 04:53:14,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-09 04:53:14,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-09 04:53:14,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-09 04:53:14,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-09 04:53:14,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-09 04:53:14,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-09 04:53:14,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-09 04:53:14,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-09 04:53:14,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-09 04:53:14,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-09 04:53:14,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-09 04:53:14,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-09 04:53:14,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-09 04:53:14,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-09 04:53:14,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-09 04:53:14,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-09 04:53:14,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-09 04:53:14,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-09 04:53:14,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-09 04:53:14,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-09 04:53:14,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-09 04:53:14,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-09 04:53:14,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-09 04:53:14,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-09 04:53:14,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-09 04:53:14,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-09 04:53:14,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-09 04:53:14,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-09 04:53:14,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-09 04:53:14,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-09 04:53:14,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-09 04:53:14,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-09 04:53:14,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-09 04:53:14,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-09 04:53:14,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-09 04:53:14,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-09 04:53:14,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-09 04:53:14,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-09 04:53:14,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-09 04:53:14,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-09 04:53:14,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-09 04:53:14,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-09 04:53:14,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-09 04:53:14,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-09 04:53:14,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-09 04:53:14,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-09 04:53:14,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-09 04:53:14,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-09 04:53:14,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-09 04:53:14,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-09 04:53:14,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-09 04:53:14,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-09 04:53:14,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-09 04:53:14,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-09 04:53:14,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-09 04:53:14,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-09 04:53:14,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-09 04:53:14,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-09 04:53:14,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-09 04:53:14,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-09 04:53:14,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-09 04:53:14,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-09 04:53:14,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-09 04:53:14,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-09 04:53:14,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-09 04:53:14,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-09 04:53:14,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-09 04:53:14,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-09 04:53:14,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-09 04:53:14,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-09 04:53:14,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-09 04:53:14,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-09 04:53:14,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-09 04:53:14,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-09 04:53:14,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-09 04:53:14,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-09 04:53:14,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-09 04:53:14,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-09 04:53:14,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-09 04:53:14,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-09 04:53:14,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-09 04:53:14,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-09 04:53:14,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-09 04:53:14,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-09 04:53:14,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-09 04:53:14,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-09 04:53:14,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-09 04:53:14,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-09 04:53:14,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-09 04:53:14,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-09 04:53:14,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-09 04:53:14,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-09 04:53:14,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-09 04:53:14,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-09 04:53:14,158 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.9433327711410975E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-09 04:53:14,158 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1546731300:82 , srv1895355361:83 , srv259299134:83 , srv496163075:83 , srv629975555:83 , srv740310720:83 , srv1059973772:84 , srv1129698475:84 , srv114800282:84 , srv1151005220:84 , srv1193665016:84 , srv1265594066:84 , srv1287033245:84 , srv1293510105:84 , srv1342618209:84 , srv1386995675:84 , srv1412206937:84 , srv1427597009:84 , srv1441095568:84 , srv1562422399:84 , srv158038689:84 , srv1689816081:84 , srv1784061022:84 , srv1790653471:84 , srv1823196916:84 , srv1832023397:84 , srv1904218294:84 , srv199422394:84 , srv2002481733:84 , srv2050380710:84 , srv214041005:84 , srv267942587:84 , srv286971509:84 , srv377189173:84 , srv377902388:84 , srv396533439:84 , srv404206040:84 , srv467377169:84 , srv531785811:84 , srv563058470:84 , srv632769746:84 , srv744023651:84 , srv775709120:84 , srv960004037:84 , srv967654647:84 , srv1018623635:85 , srv106769051:85 , srv112156127:85 , srv135721240:85 , srv1358852821:85 , srv1432817660:85 , srv1453720848:85 , srv1477772591:85 , srv1496887042:85 , srv151650185:85 , srv1540643053:85 , srv1556828771:85 , srv1586952929:85 , srv16224992:85 , srv1630578377:85 , srv1646687460:85 , srv1684555663:85 , srv1697014852:85 , srv1710417897:85 , srv1773045589:85 , srv1885132761:85 , srv1903300044:85 , srv1905572029:85 , srv1921948446:85 , srv2131287921:85 , srv213674544:85 , srv321182971:85 , srv32627012:85 , srv358635453:85 , srv358810055:85 , srv472328205:85 , srv489047925:85 , srv514951700:85 , srv545116235:85 , srv580692096:85 , srv658580786:85 , srv658865334:85 , srv720165865:85 , srv727016602:85 , srv74482232:85 , srv758361957:85 , srv805971259:85 , srv823179551:85 , srv830633234:85 , srv857731699:85 , srv874037811:85 , srv908950289:85 , srv917172563:85 , srv976266330:85 , srv1417968277:86 , srv1578833847:86 , srv1685549444:86 , srv409657778:86 , srv475972221:86 , srv352160384:632 }
2023-08-09 04:53:14,202 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=246 (was 246), ProcessCount=174 (was 174), AvailableMemoryMB=1670 (was 2529)
2023-08-09 04:53:14,213 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=246, ProcessCount=174, AvailableMemoryMB=1670
2023-08-09 04:53:14,214 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-08-09 04:53:14,214 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-08-09 04:53:14,232 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1028482939:84 , srv1046204234:84 , srv1052920546:84 , srv115032668:84 , srv132618252:84 , srv1336783583:84 , srv1342175855:84 , srv1379646490:84 , srv1394823589:84 , srv1411458040:84 , srv1616381513:84 , srv1651870277:84 , srv171506005:84 , srv18348319:84 , srv1867340683:84 , srv1988127579:84 , srv2022237364:84 , srv2027242931:84 , srv203802161:84 , srv2039763369:84 , srv2141148804:84 , srv236933382:84 , srv246084456:84 , srv383180951:84 , srv680339505:84 , srv71208375:84 , srv713016121:84 , srv903335400:84 , srv944868434:84 , srv1452485242:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-08-09 04:53:14,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-09 04:53:14,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1452485242=10, srv1988127579=16, srv1342175855=6, srv2039763369=20, srv236933382=22, srv246084456=23, srv1028482939=0, srv1616381513=11, srv18348319=14, srv680339505=25, srv132618252=4, srv71208375=26, srv1046204234=1, srv1867340683=15, srv1379646490=7, srv944868434=29, srv713016121=27, srv903335400=28, srv1052920546=2, srv1651870277=12, srv2027242931=18, srv383180951=24, srv2022237364=17, srv1411458040=9, srv2141148804=21, srv115032668=3, srv203802161=19, srv1394823589=8, srv171506005=13, srv1336783583=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-09 04:53:14,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-09 04:53:14,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-09 04:53:14,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-09 04:53:14,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-09 04:53:14,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-09 04:53:14,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-09 04:53:14,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-09 04:53:14,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-09 04:53:14,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-09 04:53:14,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-09 04:53:14,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-09 04:53:14,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-09 04:53:14,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-09 04:53:14,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-09 04:53:14,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-09 04:53:14,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-09 04:53:14,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-09 04:53:14,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-09 04:53:14,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-09 04:53:14,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-09 04:53:14,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-09 04:53:14,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-09 04:53:14,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-09 04:53:14,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-09 04:53:14,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-09 04:53:14,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-09 04:53:14,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-09 04:53:14,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-09 04:53:14,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-09 04:53:14,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-09 04:53:14,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-08-09 04:53:14,276 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0974287626462182, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-09 04:53:14,276 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912500038114037, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0974287626462182, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-08-09 04:55:14,255 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6335291 different iterations.  Found a solution that moves 1851 regions; Going from a computed imbalance of 0.9912500038114037 to a new imbalance of 4.320326340186438E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6855555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-09 04:55:14,261 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1028482939:90 , srv1046204234:90 , srv1052920546:90 , srv115032668:90 , srv132618252:90 , srv1336783583:90 , srv1342175855:90 , srv1379646490:90 , srv1394823589:90 , srv1411458040:90 , srv1452485242:90 , srv1616381513:90 , srv1651870277:90 , srv171506005:90 , srv18348319:90 , srv1867340683:90 , srv1988127579:90 , srv2022237364:90 , srv2027242931:90 , srv203802161:90 , srv2039763369:90 , srv2141148804:90 , srv236933382:90 , srv246084456:90 , srv383180951:90 , srv680339505:90 , srv71208375:90 , srv713016121:90 , srv903335400:90 , srv944868434:90 }
2023-08-09 04:55:14,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-09 04:55:14,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1452485242=10, srv1988127579=16, srv1342175855=6, srv2039763369=20, srv236933382=22, srv246084456=23, srv1028482939=0, srv1616381513=11, srv18348319=14, srv680339505=25, srv132618252=4, srv71208375=26, srv1046204234=1, srv1867340683=15, srv1379646490=7, srv944868434=29, srv713016121=27, srv903335400=28, srv1052920546=2, srv1651870277=12, srv2027242931=18, srv383180951=24, srv2022237364=17, srv1411458040=9, srv2141148804=21, srv115032668=3, srv203802161=19, srv1394823589=8, srv171506005=13, srv1336783583=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-09 04:55:14,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-09 04:55:14,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-09 04:55:14,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-09 04:55:14,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-09 04:55:14,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-09 04:55:14,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-09 04:55:14,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-09 04:55:14,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-09 04:55:14,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-09 04:55:14,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-09 04:55:14,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-09 04:55:14,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-09 04:55:14,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-09 04:55:14,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-09 04:55:14,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-09 04:55:14,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-09 04:55:14,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-09 04:55:14,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-09 04:55:14,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-09 04:55:14,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-09 04:55:14,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-09 04:55:14,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-09 04:55:14,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-09 04:55:14,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-09 04:55:14,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-09 04:55:14,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-09 04:55:14,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-09 04:55:14,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-09 04:55:14,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-09 04:55:14,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-09 04:55:14,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-08-09 04:55:14,278 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-09 04:55:14,279 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1028482939:90 , srv1046204234:90 , srv1052920546:90 , srv115032668:90 , srv132618252:90 , srv1336783583:90 , srv1342175855:90 , srv1379646490:90 , srv1394823589:90 , srv1411458040:90 , srv1452485242:90 , srv1616381513:90 , srv1651870277:90 , srv171506005:90 , srv18348319:90 , srv1867340683:90 , srv1988127579:90 , srv2022237364:90 , srv2027242931:90 , srv203802161:90 , srv2039763369:90 , srv2141148804:90 , srv236933382:90 , srv246084456:90 , srv383180951:90 , srv680339505:90 , srv71208375:90 , srv713016121:90 , srv903335400:90 , srv944868434:90 }
2023-08-09 04:55:14,291 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=225 (was 246), ProcessCount=171 (was 174), AvailableMemoryMB=1561 (was 1670)