2023-07-13 14:53:10,068 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-07-13 14:53:10,394 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-07-13 14:53:10,512 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-07-13 14:53:10,577 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-07-13 14:53:10,586 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-07-13 14:53:10,603 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=365, ProcessCount=172, AvailableMemoryMB=8730
2023-07-13 14:53:10,605 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-07-13 14:53:10,605 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-07-13 14:53:10,765 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1015443873:84 , srv1025042129:84 , srv1059344535:84 , srv1060755587:84 , srv1117232329:84 , srv1155923280:84 , srv1182903868:84 , srv120340867:84 , srv1226466844:84 , srv1232469874:84 , srv1242927135:84 , srv1249242657:84 , srv1263533770:84 , srv126392810:84 , srv1272814976:84 , srv1297557615:84 , srv1313981157:84 , srv1317373061:84 , srv1390877124:84 , srv1404390838:84 , srv1419190468:84 , srv1433140807:84 , srv1434778629:84 , srv1468396867:84 , srv1509199719:84 , srv1512604359:84 , srv1520435814:84 , srv1523466935:84 , srv1543276081:84 , srv1560263249:84 , srv1569800648:84 , srv1570474280:84 , srv1572540500:84 , srv1596371857:84 , srv1621212261:84 , srv164418949:84 , srv1690810884:84 , srv1694375937:84 , srv1717750042:84 , srv1729945837:84 , srv1754317133:84 , srv1764106316:84 , srv177382926:84 , srv1798169692:84 , srv1803882806:84 , srv1814574887:84 , srv1822670328:84 , srv1833146081:84 , srv1845234155:84 , srv18684113:84 , srv1904097318:84 , srv1916962882:84 , srv1943112758:84 , srv1960140168:84 , srv1996319694:84 , srv2010258547:84 , srv2036167029:84 , srv2040656956:84 , srv2085926014:84 , srv2111619928:84 , srv2116271781:84 , srv2124705356:84 , srv212705111:84 , srv2134808535:84 , srv2134824044:84 , srv231426802:84 , srv286714588:84 , srv354392353:84 , srv355704197:84 , srv366590463:84 , srv368342418:84 , srv370755947:84 , srv444856620:84 , srv507789137:84 , srv517905361:84 , srv52128557:84 , srv526321780:84 , srv541167176:84 , srv547317728:84 , srv618661760:84 , srv633919242:84 , srv638118987:84 , srv638523012:84 , srv667501064:84 , srv685895590:84 , srv693234450:84 , srv705498543:84 , srv711491679:84 , srv741553599:84 , srv779845732:84 , srv82121537:84 , srv823812947:84 , srv860829113:84 , srv863713138:84 , srv901350315:84 , srv912914686:84 , srv935954853:84 , srv94857398:84 , srv996068448:84 , srv15162245:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-07-13 14:53:10,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-13 14:53:10,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv860829113=93, srv1297557615=15, srv177382926=43, srv1242927135=10, srv741553599=89, srv517905361=75, srv1272814976=14, srv212705111=63, srv1621212261=35, srv1764106316=42, srv1904097318=51, srv1833146081=48, srv901350315=95, srv1313981157=16, srv1996319694=55, srv82121537=91, srv2085926014=59, srv685895590=85, srv638118987=82, srv1690810884=37, srv120340867=7, srv18684113=50, srv1543276081=29, srv94857398=98, srv1015443873=0, srv863713138=94, srv1232469874=9, srv354392353=68, srv912914686=96, srv1569800648=31, srv667501064=84, srv1694375937=38, srv1249242657=11, srv1596371857=34, srv1434778629=22, srv1117232329=4, srv1560263249=30, srv547317728=79, srv1512604359=25, srv1960140168=54, srv633919242=81, srv711491679=88, srv1390877124=18, srv1059344535=2, srv1509199719=24, srv444856620=73, srv996068448=99, srv1570474280=32, srv638523012=83, srv355704197=69, srv1468396867=23, srv2040656956=58, srv2124705356=62, srv1520435814=27, srv2010258547=56, srv1523466935=28, srv1572540500=33, srv1798169692=44, srv1182903868=6, srv823812947=92, srv1916962882=52, srv1754317133=41, srv541167176=78, srv366590463=70, srv507789137=74, srv779845732=90, srv1404390838=19, srv705498543=87, srv1814574887=46, srv370755947=72, srv1263533770=12, srv1845234155=49, srv2036167029=57, srv618661760=80, srv693234450=86, srv1060755587=3, srv2111619928=60, srv1226466844=8, srv1155923280=5, srv1943112758=53, srv526321780=77, srv935954853=97, srv126392810=13, srv1433140807=21, srv1717750042=39, srv368342418=71, srv2116271781=61, srv1803882806=45, srv2134824044=65, srv1729945837=40, srv52128557=76, srv286714588=67, srv1025042129=1, srv231426802=66, srv15162245=26, srv2134808535=64, srv1317373061=17, srv1822670328=47, srv1419190468=20, srv164418949=36} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-13 14:53:10,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-13 14:53:10,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-13 14:53:10,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-13 14:53:10,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-13 14:53:10,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-13 14:53:10,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-13 14:53:10,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-13 14:53:10,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-13 14:53:10,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-13 14:53:10,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-13 14:53:10,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-13 14:53:10,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-13 14:53:10,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-13 14:53:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-13 14:53:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-13 14:53:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-13 14:53:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-13 14:53:10,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-13 14:53:10,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-13 14:53:10,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-13 14:53:10,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-13 14:53:10,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-13 14:53:10,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-13 14:53:10,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-13 14:53:10,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-13 14:53:10,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-13 14:53:10,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-13 14:53:10,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-13 14:53:10,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-13 14:53:10,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-13 14:53:10,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-13 14:53:10,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-13 14:53:10,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-13 14:53:10,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-13 14:53:10,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-13 14:53:10,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-13 14:53:10,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-13 14:53:10,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-13 14:53:10,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-13 14:53:10,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-13 14:53:10,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-13 14:53:10,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-13 14:53:10,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-13 14:53:10,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-13 14:53:10,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-13 14:53:10,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-13 14:53:10,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-13 14:53:10,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-13 14:53:10,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-13 14:53:10,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-13 14:53:10,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-13 14:53:10,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-13 14:53:10,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-13 14:53:10,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-13 14:53:10,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-13 14:53:10,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-13 14:53:10,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-13 14:53:10,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-13 14:53:10,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-13 14:53:10,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-13 14:53:10,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-13 14:53:10,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-13 14:53:10,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-13 14:53:10,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-13 14:53:10,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-13 14:53:10,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-13 14:53:10,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-13 14:53:10,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-13 14:53:10,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-13 14:53:10,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-13 14:53:10,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-13 14:53:10,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-13 14:53:10,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-13 14:53:10,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-13 14:53:10,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-13 14:53:10,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-13 14:53:10,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-13 14:53:10,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-13 14:53:10,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-13 14:53:10,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-13 14:53:10,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-13 14:53:10,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-13 14:53:10,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-13 14:53:10,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-13 14:53:10,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-13 14:53:10,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-13 14:53:10,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-13 14:53:10,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-13 14:53:10,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-13 14:53:10,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-13 14:53:10,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-13 14:53:10,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-13 14:53:10,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-13 14:53:10,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-13 14:53:10,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-13 14:53:10,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-13 14:53:10,896 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-13 14:53:10,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-13 14:53:10,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-13 14:53:10,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-13 14:53:10,897 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-13 14:53:11,150 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-13 14:53:11,150 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-07-13 14:53:11,151 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-07-13 14:53:11,583 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 665 ms to try 5000 different iterations.  Found a solution that moves 4681 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18451953727401096. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03827265484965151, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055186653992610496, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5201111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03827265484965151, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24441666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-13 14:53:11,613 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1317373061:81 , srv1232469874:82 , srv1560263249:82 , srv1570474280:82 , srv1833146081:82 , srv667501064:82 , srv1313981157:83 , srv1433140807:83 , srv1468396867:83 , srv2124705356:83 , srv231426802:83 , srv1025042129:84 , srv1543276081:84 , srv164418949:84 , srv1845234155:84 , srv2036167029:84 , srv2116271781:84 , srv354392353:84 , srv370755947:84 , srv996068448:84 , srv1015443873:85 , srv1060755587:85 , srv1182903868:85 , srv1263533770:85 , srv1272814976:85 , srv1404390838:85 , srv1509199719:85 , srv1621212261:85 , srv1690810884:85 , srv1754317133:85 , srv1798169692:85 , srv2010258547:85 , srv2134808535:85 , srv541167176:85 , srv685895590:85 , srv741553599:85 , srv1249242657:86 , srv1419190468:86 , srv1512604359:86 , srv1569800648:86 , srv1596371857:86 , srv1694375937:86 , srv177382926:86 , srv286714588:86 , srv444856620:86 , srv633919242:86 , srv638118987:86 , srv638523012:86 , srv779845732:86 , srv860829113:86 , srv901350315:86 , srv912914686:86 , srv94857398:86 , srv1117232329:87 , srv1434778629:87 , srv1717750042:87 , srv1764106316:87 , srv1996319694:87 , srv355704197:87 , srv366590463:87 , srv368342418:87 , srv507789137:87 , srv82121537:87 , srv863713138:87 , srv1297557615:88 , srv1572540500:88 , srv1729945837:88 , srv1814574887:88 , srv18684113:88 , srv1904097318:88 , srv1943112758:88 , srv2085926014:88 , srv2111619928:88 , srv212705111:88 , srv618661760:88 , srv693234450:88 , srv1155923280:89 , srv120340867:89 , srv1242927135:89 , srv126392810:89 , srv1520435814:89 , srv1523466935:89 , srv1803882806:89 , srv1822670328:89 , srv1916962882:89 , srv1960140168:89 , srv2040656956:89 , srv547317728:89 , srv711491679:89 , srv823812947:89 , srv517905361:90 , srv526321780:90 , srv935954853:90 , srv1390877124:91 , srv2134824044:91 , srv1226466844:93 , srv1059344535:94 , srv52128557:94 , srv705498543:94 , srv15162245:430 }
2023-07-13 14:53:11,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-13 14:53:11,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv860829113=93, srv1297557615=15, srv177382926=43, srv1242927135=10, srv741553599=89, srv517905361=75, srv1272814976=14, srv212705111=63, srv1621212261=35, srv1764106316=42, srv1904097318=51, srv1833146081=48, srv901350315=95, srv1313981157=16, srv1996319694=55, srv82121537=91, srv2085926014=59, srv685895590=85, srv638118987=82, srv1690810884=37, srv120340867=7, srv18684113=50, srv1543276081=29, srv94857398=98, srv1015443873=0, srv863713138=94, srv1232469874=9, srv354392353=68, srv912914686=96, srv1569800648=31, srv667501064=84, srv1694375937=38, srv1249242657=11, srv1596371857=34, srv1434778629=22, srv1117232329=4, srv1560263249=30, srv547317728=79, srv1512604359=25, srv1960140168=54, srv633919242=81, srv711491679=88, srv1390877124=18, srv1059344535=2, srv1509199719=24, srv444856620=73, srv996068448=99, srv1570474280=32, srv638523012=83, srv355704197=69, srv1468396867=23, srv2040656956=58, srv2124705356=62, srv1520435814=27, srv2010258547=56, srv1523466935=28, srv1572540500=33, srv1798169692=44, srv1182903868=6, srv823812947=92, srv1916962882=52, srv1754317133=41, srv541167176=78, srv366590463=70, srv507789137=74, srv779845732=90, srv1404390838=19, srv705498543=87, srv1814574887=46, srv370755947=72, srv1263533770=12, srv1845234155=49, srv2036167029=57, srv618661760=80, srv693234450=86, srv1060755587=3, srv2111619928=60, srv1226466844=8, srv1155923280=5, srv1943112758=53, srv526321780=77, srv935954853=97, srv126392810=13, srv1433140807=21, srv1717750042=39, srv368342418=71, srv2116271781=61, srv1803882806=45, srv2134824044=65, srv1729945837=40, srv52128557=76, srv286714588=67, srv1025042129=1, srv231426802=66, srv15162245=26, srv2134808535=64, srv1317373061=17, srv1822670328=47, srv1419190468=20, srv164418949=36} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-13 14:53:11,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-13 14:53:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-13 14:53:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-13 14:53:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-13 14:53:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-13 14:53:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-13 14:53:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-13 14:53:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-13 14:53:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-13 14:53:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-13 14:53:11,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-13 14:53:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-13 14:53:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-13 14:53:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-13 14:53:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-13 14:53:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-13 14:53:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-13 14:53:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-13 14:53:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-13 14:53:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-13 14:53:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-13 14:53:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-13 14:53:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-13 14:53:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-13 14:53:11,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-13 14:53:11,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-13 14:53:11,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-13 14:53:11,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-13 14:53:11,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-13 14:53:11,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-13 14:53:11,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-13 14:53:11,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-13 14:53:11,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-13 14:53:11,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-13 14:53:11,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-13 14:53:11,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-13 14:53:11,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-13 14:53:11,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-13 14:53:11,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-13 14:53:11,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-13 14:53:11,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-13 14:53:11,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-13 14:53:11,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-13 14:53:11,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-13 14:53:11,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-13 14:53:11,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-13 14:53:11,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-13 14:53:11,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-13 14:53:11,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-13 14:53:11,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-13 14:53:11,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-13 14:53:11,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-13 14:53:11,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-13 14:53:11,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-13 14:53:11,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-13 14:53:11,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-13 14:53:11,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-13 14:53:11,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-13 14:53:11,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-13 14:53:11,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-13 14:53:11,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-13 14:53:11,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-13 14:53:11,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-13 14:53:11,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-13 14:53:11,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-13 14:53:11,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-13 14:53:11,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-13 14:53:11,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-13 14:53:11,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-13 14:53:11,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-13 14:53:11,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-13 14:53:11,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-13 14:53:11,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-13 14:53:11,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-13 14:53:11,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-13 14:53:11,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-13 14:53:11,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-13 14:53:11,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-13 14:53:11,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-13 14:53:11,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-13 14:53:11,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-13 14:53:11,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-13 14:53:11,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-13 14:53:11,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-13 14:53:11,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-13 14:53:11,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-13 14:53:11,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-13 14:53:11,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-13 14:53:11,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-13 14:53:11,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-13 14:53:11,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-13 14:53:11,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-13 14:53:11,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-13 14:53:11,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-13 14:53:11,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-13 14:53:11,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-13 14:53:11,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-13 14:53:11,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-13 14:53:11,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-13 14:53:11,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-13 14:53:11,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-13 14:53:11,851 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03827265484965151, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055186653992610496, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03827265484965151, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24441666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-13 14:53:11,851 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-07-13 14:53:11,851 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18448676021145274, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03827265484965151, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055186653992610496, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03827265484965151, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24441666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-07-13 14:53:12,230 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 556 ms to try 5000 different iterations.  Found a solution that moves 2614 regions; Going from a computed imbalance of 0.18448676021145274 to a new imbalance of 0.016408077224270283. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021163323701831247); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035075675764025274, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29044444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021163323701831247); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-13 14:53:12,244 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv52128557:77 , srv1059344535:79 , srv1226466844:79 , srv705498543:79 , srv120340867:81 , srv1523466935:81 , srv1242927135:82 , srv1390877124:82 , srv1520435814:82 , srv1814574887:82 , srv2040656956:82 , srv126392810:83 , srv1822670328:83 , srv18684113:83 , srv1916962882:83 , srv1943112758:83 , srv1960140168:83 , srv2085926014:83 , srv2134824044:83 , srv368342418:83 , srv633919242:83 , srv711491679:83 , srv823812947:83 , srv1419190468:84 , srv1434778629:84 , srv1572540500:84 , srv1717750042:84 , srv1729945837:84 , srv1764106316:84 , srv1803882806:84 , srv212705111:84 , srv355704197:84 , srv366590463:84 , srv517905361:84 , srv526321780:84 , srv779845732:84 , srv1404390838:85 , srv1694375937:85 , srv177382926:85 , srv1904097318:85 , srv1996319694:85 , srv2134808535:85 , srv507789137:85 , srv863713138:85 , srv912914686:85 , srv1117232329:86 , srv1155923280:86 , srv1182903868:86 , srv1297557615:86 , srv1543276081:86 , srv1569800648:86 , srv1596371857:86 , srv2111619928:86 , srv444856620:86 , srv541167176:86 , srv547317728:86 , srv618661760:86 , srv638118987:86 , srv82121537:86 , srv901350315:86 , srv935954853:86 , srv1249242657:87 , srv1263533770:87 , srv1509199719:87 , srv1512604359:87 , srv1621212261:87 , srv1690810884:87 , srv1798169692:87 , srv2010258547:87 , srv2116271781:87 , srv286714588:87 , srv860829113:87 , srv1015443873:88 , srv1025042129:88 , srv1272814976:88 , srv1313981157:88 , srv1468396867:88 , srv1570474280:88 , srv1754317133:88 , srv1845234155:88 , srv354392353:88 , srv370755947:88 , srv693234450:88 , srv741553599:88 , srv94857398:88 , srv1060755587:89 , srv1232469874:89 , srv1317373061:89 , srv1560263249:89 , srv164418949:89 , srv1833146081:89 , srv2036167029:89 , srv2124705356:89 , srv667501064:89 , srv1433140807:90 , srv231426802:90 , srv685895590:90 , srv996068448:90 , srv638523012:91 , srv15162245:532 }
2023-07-13 14:53:12,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-13 14:53:12,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv860829113=93, srv1297557615=15, srv177382926=43, srv1242927135=10, srv741553599=89, srv517905361=75, srv1272814976=14, srv212705111=63, srv1621212261=35, srv1764106316=42, srv1904097318=51, srv1833146081=48, srv901350315=95, srv1313981157=16, srv1996319694=55, srv82121537=91, srv2085926014=59, srv685895590=85, srv638118987=82, srv1690810884=37, srv120340867=7, srv18684113=50, srv1543276081=29, srv94857398=98, srv1015443873=0, srv863713138=94, srv1232469874=9, srv354392353=68, srv912914686=96, srv1569800648=31, srv667501064=84, srv1694375937=38, srv1249242657=11, srv1596371857=34, srv1434778629=22, srv1117232329=4, srv1560263249=30, srv547317728=79, srv1512604359=25, srv1960140168=54, srv633919242=81, srv711491679=88, srv1390877124=18, srv1059344535=2, srv1509199719=24, srv444856620=73, srv996068448=99, srv1570474280=32, srv638523012=83, srv355704197=69, srv1468396867=23, srv2040656956=58, srv2124705356=62, srv1520435814=27, srv2010258547=56, srv1523466935=28, srv1572540500=33, srv1798169692=44, srv1182903868=6, srv823812947=92, srv1916962882=52, srv1754317133=41, srv541167176=78, srv366590463=70, srv507789137=74, srv779845732=90, srv1404390838=19, srv705498543=87, srv1814574887=46, srv370755947=72, srv1263533770=12, srv1845234155=49, srv2036167029=57, srv618661760=80, srv693234450=86, srv1060755587=3, srv2111619928=60, srv1226466844=8, srv1155923280=5, srv1943112758=53, srv526321780=77, srv935954853=97, srv126392810=13, srv1433140807=21, srv1717750042=39, srv368342418=71, srv2116271781=61, srv1803882806=45, srv2134824044=65, srv1729945837=40, srv52128557=76, srv286714588=67, srv1025042129=1, srv231426802=66, srv15162245=26, srv2134808535=64, srv1317373061=17, srv1822670328=47, srv1419190468=20, srv164418949=36} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-13 14:53:12,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-13 14:53:12,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-13 14:53:12,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-13 14:53:12,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-13 14:53:12,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-13 14:53:12,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-13 14:53:12,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-13 14:53:12,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-13 14:53:12,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-13 14:53:12,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-13 14:53:12,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-13 14:53:12,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-13 14:53:12,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-13 14:53:12,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-13 14:53:12,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-13 14:53:12,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-13 14:53:12,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-13 14:53:12,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-13 14:53:12,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-13 14:53:12,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-13 14:53:12,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-13 14:53:12,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-13 14:53:12,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-13 14:53:12,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-13 14:53:12,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-13 14:53:12,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-13 14:53:12,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-13 14:53:12,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-13 14:53:12,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-13 14:53:12,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-13 14:53:12,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-13 14:53:12,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-13 14:53:12,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-13 14:53:12,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-13 14:53:12,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-13 14:53:12,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-13 14:53:12,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-13 14:53:12,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-13 14:53:12,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-13 14:53:12,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-13 14:53:12,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-13 14:53:12,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-13 14:53:12,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-13 14:53:12,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-13 14:53:12,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-13 14:53:12,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-13 14:53:12,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-13 14:53:12,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-13 14:53:12,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-13 14:53:12,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-13 14:53:12,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-13 14:53:12,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-13 14:53:12,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-13 14:53:12,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-13 14:53:12,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-13 14:53:12,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-13 14:53:12,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-13 14:53:12,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-13 14:53:12,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-13 14:53:12,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-13 14:53:12,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-13 14:53:12,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-13 14:53:12,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-13 14:53:12,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-13 14:53:12,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-13 14:53:12,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-13 14:53:12,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-13 14:53:12,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-13 14:53:12,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-13 14:53:12,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-13 14:53:12,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-13 14:53:12,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-13 14:53:12,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-13 14:53:12,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-13 14:53:12,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-13 14:53:12,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-13 14:53:12,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-13 14:53:12,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-13 14:53:12,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-13 14:53:12,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-13 14:53:12,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-13 14:53:12,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-13 14:53:12,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-13 14:53:12,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-13 14:53:12,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-13 14:53:12,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-13 14:53:12,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-13 14:53:12,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-13 14:53:12,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-13 14:53:12,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-13 14:53:12,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-13 14:53:12,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-13 14:53:12,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-13 14:53:12,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-13 14:53:12,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-13 14:53:12,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-13 14:53:12,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-13 14:53:12,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-13 14:53:12,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-13 14:53:12,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-13 14:53:12,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-13 14:53:12,439 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021163323701831247); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035075675764025274, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021163323701831247); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-13 14:53:12,440 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-07-13 14:53:12,440 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016389773605059183, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021163323701831247); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035075675764025274, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021163323701831247); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-07-13 14:53:12,747 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 440 ms to try 5000 different iterations.  Found a solution that moves 952 regions; Going from a computed imbalance of 0.016389773605059183 to a new imbalance of 8.335517883202082E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005105203241514896); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00823340762517771); MoveCostFunction : (multiplier=7.0, imbalance=0.10577777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005105203241514896); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-13 14:53:12,749 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv638523012:80 , srv693234450:82 , srv935954853:82 , srv1060755587:83 , srv1155923280:83 , srv2111619928:83 , srv286714588:83 , srv685895590:83 , srv94857398:83 , srv996068448:83 , srv1015443873:84 , srv1117232329:84 , srv1249242657:84 , srv1297557615:84 , srv1512604359:84 , srv164418949:84 , srv1754317133:84 , srv1904097318:84 , srv2134824044:84 , srv517905361:84 , srv526321780:84 , srv547317728:84 , srv618661760:84 , srv705498543:84 , srv741553599:84 , srv82121537:84 , srv860829113:84 , srv1025042129:85 , srv1059344535:85 , srv1263533770:85 , srv126392810:85 , srv1272814976:85 , srv1433140807:85 , srv1509199719:85 , srv1596371857:85 , srv1729945837:85 , srv1798169692:85 , srv1822670328:85 , srv1916962882:85 , srv2036167029:85 , srv2124705356:85 , srv212705111:85 , srv711491679:85 , srv863713138:85 , srv901350315:85 , srv1182903868:86 , srv1226466844:86 , srv1242927135:86 , srv1390877124:86 , srv1468396867:86 , srv1520435814:86 , srv1560263249:86 , srv1569800648:86 , srv1690810884:86 , srv1717750042:86 , srv1764106316:86 , srv177382926:86 , srv1803882806:86 , srv1833146081:86 , srv1845234155:86 , srv18684113:86 , srv1943112758:86 , srv1996319694:86 , srv2010258547:86 , srv2040656956:86 , srv2085926014:86 , srv2116271781:86 , srv231426802:86 , srv354392353:86 , srv355704197:86 , srv366590463:86 , srv444856620:86 , srv507789137:86 , srv541167176:86 , srv638118987:86 , srv823812947:86 , srv1232469874:87 , srv1313981157:87 , srv1317373061:87 , srv1404390838:87 , srv1434778629:87 , srv1572540500:87 , srv1621212261:87 , srv1694375937:87 , srv1814574887:87 , srv1960140168:87 , srv370755947:87 , srv52128557:87 , srv667501064:87 , srv779845732:87 , srv912914686:87 , srv120340867:88 , srv1419190468:88 , srv1543276081:88 , srv1570474280:88 , srv2134808535:88 , srv368342418:88 , srv633919242:88 , srv1523466935:89 , srv15162245:541 }
2023-07-13 14:53:12,753 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-13 14:53:12,754 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv860829113=93, srv1297557615=15, srv177382926=43, srv1242927135=10, srv741553599=89, srv517905361=75, srv1272814976=14, srv212705111=63, srv1621212261=35, srv1764106316=42, srv1904097318=51, srv1833146081=48, srv901350315=95, srv1313981157=16, srv1996319694=55, srv82121537=91, srv2085926014=59, srv685895590=85, srv638118987=82, srv1690810884=37, srv120340867=7, srv18684113=50, srv1543276081=29, srv94857398=98, srv1015443873=0, srv863713138=94, srv1232469874=9, srv354392353=68, srv912914686=96, srv1569800648=31, srv667501064=84, srv1694375937=38, srv1249242657=11, srv1596371857=34, srv1434778629=22, srv1117232329=4, srv1560263249=30, srv547317728=79, srv1512604359=25, srv1960140168=54, srv633919242=81, srv711491679=88, srv1390877124=18, srv1059344535=2, srv1509199719=24, srv444856620=73, srv996068448=99, srv1570474280=32, srv638523012=83, srv355704197=69, srv1468396867=23, srv2040656956=58, srv2124705356=62, srv1520435814=27, srv2010258547=56, srv1523466935=28, srv1572540500=33, srv1798169692=44, srv1182903868=6, srv823812947=92, srv1916962882=52, srv1754317133=41, srv541167176=78, srv366590463=70, srv507789137=74, srv779845732=90, srv1404390838=19, srv705498543=87, srv1814574887=46, srv370755947=72, srv1263533770=12, srv1845234155=49, srv2036167029=57, srv618661760=80, srv693234450=86, srv1060755587=3, srv2111619928=60, srv1226466844=8, srv1155923280=5, srv1943112758=53, srv526321780=77, srv935954853=97, srv126392810=13, srv1433140807=21, srv1717750042=39, srv368342418=71, srv2116271781=61, srv1803882806=45, srv2134824044=65, srv1729945837=40, srv52128557=76, srv286714588=67, srv1025042129=1, srv231426802=66, srv15162245=26, srv2134808535=64, srv1317373061=17, srv1822670328=47, srv1419190468=20, srv164418949=36} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-13 14:53:12,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-13 14:53:12,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-13 14:53:12,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-13 14:53:12,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-13 14:53:12,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-13 14:53:12,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-13 14:53:12,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-13 14:53:12,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-13 14:53:12,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-13 14:53:12,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-13 14:53:12,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-13 14:53:12,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-13 14:53:12,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-13 14:53:12,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-13 14:53:12,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-13 14:53:12,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-13 14:53:12,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-13 14:53:12,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-13 14:53:12,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-13 14:53:12,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-13 14:53:12,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-13 14:53:12,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-13 14:53:12,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-13 14:53:12,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-13 14:53:12,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-13 14:53:12,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-13 14:53:12,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-13 14:53:12,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-13 14:53:12,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-13 14:53:12,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-13 14:53:12,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-13 14:53:12,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-13 14:53:12,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-13 14:53:12,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-13 14:53:12,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-13 14:53:12,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-13 14:53:12,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-13 14:53:12,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-13 14:53:12,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-13 14:53:12,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-13 14:53:12,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-13 14:53:12,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-13 14:53:12,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-13 14:53:12,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-13 14:53:12,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-13 14:53:12,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-13 14:53:12,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-13 14:53:12,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-13 14:53:12,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-13 14:53:12,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-13 14:53:12,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-13 14:53:12,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-13 14:53:12,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-13 14:53:12,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-13 14:53:12,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-13 14:53:12,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-13 14:53:12,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-13 14:53:12,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-13 14:53:12,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-13 14:53:12,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-13 14:53:12,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-13 14:53:12,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-13 14:53:12,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-13 14:53:12,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-13 14:53:12,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-13 14:53:12,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-13 14:53:12,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-13 14:53:12,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-13 14:53:12,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-13 14:53:12,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-13 14:53:12,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-13 14:53:12,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-13 14:53:12,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-13 14:53:12,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-13 14:53:12,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-13 14:53:12,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-13 14:53:12,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-13 14:53:12,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-13 14:53:12,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-13 14:53:12,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-13 14:53:12,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-13 14:53:12,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-13 14:53:12,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-13 14:53:12,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-13 14:53:12,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-13 14:53:12,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-13 14:53:12,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-13 14:53:12,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-13 14:53:12,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-13 14:53:12,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-13 14:53:12,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-13 14:53:12,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-13 14:53:12,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-13 14:53:12,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-13 14:53:12,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-13 14:53:12,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-13 14:53:12,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-13 14:53:12,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-13 14:53:12,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-13 14:53:12,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-13 14:53:12,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-13 14:53:12,929 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005105203241514896); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00823340762517771); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005105203241514896); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-13 14:53:12,930 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-07-13 14:53:12,930 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.2688574184394E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005105203241514896); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00823340762517771); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005105203241514896); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-07-13 14:53:13,278 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 481 ms to try 5000 different iterations.  Found a solution that moves 239 regions; Going from a computed imbalance of 8.2688574184394E-4 to a new imbalance of 1.2355602748923052E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.026555555555555554, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-13 14:53:13,279 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1523466935:82 , srv1059344535:83 , srv1390877124:83 , srv1572540500:83 , srv1621212261:83 , srv1803882806:83 , srv1960140168:83 , srv231426802:83 , srv370755947:83 , srv547317728:83 , srv1182903868:84 , srv120340867:84 , srv1226466844:84 , srv1232469874:84 , srv1272814976:84 , srv1297557615:84 , srv1313981157:84 , srv1419190468:84 , srv1433140807:84 , srv1434778629:84 , srv1543276081:84 , srv1569800648:84 , srv1570474280:84 , srv1690810884:84 , srv1694375937:84 , srv1798169692:84 , srv1845234155:84 , srv1996319694:84 , srv2010258547:84 , srv2036167029:84 , srv2040656956:84 , srv2124705356:84 , srv2134808535:84 , srv2134824044:84 , srv354392353:84 , srv368342418:84 , srv444856620:84 , srv507789137:84 , srv52128557:84 , srv526321780:84 , srv541167176:84 , srv618661760:84 , srv638118987:84 , srv667501064:84 , srv685895590:84 , srv693234450:84 , srv823812947:84 , srv935954853:84 , srv1015443873:85 , srv1025042129:85 , srv1060755587:85 , srv1117232329:85 , srv1155923280:85 , srv1242927135:85 , srv1249242657:85 , srv1263533770:85 , srv126392810:85 , srv1317373061:85 , srv1468396867:85 , srv1509199719:85 , srv1512604359:85 , srv1560263249:85 , srv1596371857:85 , srv164418949:85 , srv1729945837:85 , srv1754317133:85 , srv1764106316:85 , srv177382926:85 , srv1814574887:85 , srv1822670328:85 , srv1833146081:85 , srv18684113:85 , srv1904097318:85 , srv1916962882:85 , srv1943112758:85 , srv2085926014:85 , srv2111619928:85 , srv2116271781:85 , srv212705111:85 , srv366590463:85 , srv517905361:85 , srv633919242:85 , srv705498543:85 , srv711491679:85 , srv741553599:85 , srv779845732:85 , srv82121537:85 , srv860829113:85 , srv863713138:85 , srv901350315:85 , srv912914686:85 , srv94857398:85 , srv996068448:85 , srv1404390838:86 , srv1520435814:86 , srv1717750042:86 , srv286714588:86 , srv355704197:86 , srv638523012:86 , srv15162245:638 }
2023-07-13 14:53:13,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-13 14:53:13,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv860829113=93, srv1297557615=15, srv177382926=43, srv1242927135=10, srv741553599=89, srv517905361=75, srv1272814976=14, srv212705111=63, srv1621212261=35, srv1764106316=42, srv1904097318=51, srv1833146081=48, srv901350315=95, srv1313981157=16, srv1996319694=55, srv82121537=91, srv2085926014=59, srv685895590=85, srv638118987=82, srv1690810884=37, srv120340867=7, srv18684113=50, srv1543276081=29, srv94857398=98, srv1015443873=0, srv863713138=94, srv1232469874=9, srv354392353=68, srv912914686=96, srv1569800648=31, srv667501064=84, srv1694375937=38, srv1249242657=11, srv1596371857=34, srv1434778629=22, srv1117232329=4, srv1560263249=30, srv547317728=79, srv1512604359=25, srv1960140168=54, srv633919242=81, srv711491679=88, srv1390877124=18, srv1059344535=2, srv1509199719=24, srv444856620=73, srv996068448=99, srv1570474280=32, srv638523012=83, srv355704197=69, srv1468396867=23, srv2040656956=58, srv2124705356=62, srv1520435814=27, srv2010258547=56, srv1523466935=28, srv1572540500=33, srv1798169692=44, srv1182903868=6, srv823812947=92, srv1916962882=52, srv1754317133=41, srv541167176=78, srv366590463=70, srv507789137=74, srv779845732=90, srv1404390838=19, srv705498543=87, srv1814574887=46, srv370755947=72, srv1263533770=12, srv1845234155=49, srv2036167029=57, srv618661760=80, srv693234450=86, srv1060755587=3, srv2111619928=60, srv1226466844=8, srv1155923280=5, srv1943112758=53, srv526321780=77, srv935954853=97, srv126392810=13, srv1433140807=21, srv1717750042=39, srv368342418=71, srv2116271781=61, srv1803882806=45, srv2134824044=65, srv1729945837=40, srv52128557=76, srv286714588=67, srv1025042129=1, srv231426802=66, srv15162245=26, srv2134808535=64, srv1317373061=17, srv1822670328=47, srv1419190468=20, srv164418949=36} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-13 14:53:13,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-13 14:53:13,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-13 14:53:13,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-13 14:53:13,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-13 14:53:13,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-13 14:53:13,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-13 14:53:13,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-13 14:53:13,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-13 14:53:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-13 14:53:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-13 14:53:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-13 14:53:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-13 14:53:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-13 14:53:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-13 14:53:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-13 14:53:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-13 14:53:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-13 14:53:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-13 14:53:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-13 14:53:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-13 14:53:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-13 14:53:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-13 14:53:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-13 14:53:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-13 14:53:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-13 14:53:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-13 14:53:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-13 14:53:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-13 14:53:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-13 14:53:13,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-13 14:53:13,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-07-13 14:53:13,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-07-13 14:53:13,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-07-13 14:53:13,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-07-13 14:53:13,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-07-13 14:53:13,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-07-13 14:53:13,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-07-13 14:53:13,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-07-13 14:53:13,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-07-13 14:53:13,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-07-13 14:53:13,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-07-13 14:53:13,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-07-13 14:53:13,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-07-13 14:53:13,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-07-13 14:53:13,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-07-13 14:53:13,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-07-13 14:53:13,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-07-13 14:53:13,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-07-13 14:53:13,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-07-13 14:53:13,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-07-13 14:53:13,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-07-13 14:53:13,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-07-13 14:53:13,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-07-13 14:53:13,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-07-13 14:53:13,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-07-13 14:53:13,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-07-13 14:53:13,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-07-13 14:53:13,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-07-13 14:53:13,305 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-07-13 14:53:13,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-07-13 14:53:13,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-07-13 14:53:13,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-07-13 14:53:13,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-07-13 14:53:13,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-07-13 14:53:13,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-07-13 14:53:13,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-07-13 14:53:13,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-07-13 14:53:13,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-07-13 14:53:13,306 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-07-13 14:53:13,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-07-13 14:53:13,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-07-13 14:53:13,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-07-13 14:53:13,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-07-13 14:53:13,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-07-13 14:53:13,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-07-13 14:53:13,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-07-13 14:53:13,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-07-13 14:53:13,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-07-13 14:53:13,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-07-13 14:53:13,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-07-13 14:53:13,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-07-13 14:53:13,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-07-13 14:53:13,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-07-13 14:53:13,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-07-13 14:53:13,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-07-13 14:53:13,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-07-13 14:53:13,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-07-13 14:53:13,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-07-13 14:53:13,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-07-13 14:53:13,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-07-13 14:53:13,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-07-13 14:53:13,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-07-13 14:53:13,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-07-13 14:53:13,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-07-13 14:53:13,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-07-13 14:53:13,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-07-13 14:53:13,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-07-13 14:53:13,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-07-13 14:53:13,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-07-13 14:53:13,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-07-13 14:53:13,309 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-07-13 14:53:13,457 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.0682088980196052E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-13 14:53:13,457 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1523466935:82 , srv1059344535:83 , srv1390877124:83 , srv1572540500:83 , srv1621212261:83 , srv1803882806:83 , srv1960140168:83 , srv231426802:83 , srv370755947:83 , srv547317728:83 , srv1182903868:84 , srv120340867:84 , srv1226466844:84 , srv1232469874:84 , srv1272814976:84 , srv1297557615:84 , srv1313981157:84 , srv1419190468:84 , srv1433140807:84 , srv1434778629:84 , srv1543276081:84 , srv1569800648:84 , srv1570474280:84 , srv1690810884:84 , srv1694375937:84 , srv1798169692:84 , srv1845234155:84 , srv1996319694:84 , srv2010258547:84 , srv2036167029:84 , srv2040656956:84 , srv2124705356:84 , srv2134808535:84 , srv2134824044:84 , srv354392353:84 , srv368342418:84 , srv444856620:84 , srv507789137:84 , srv52128557:84 , srv526321780:84 , srv541167176:84 , srv618661760:84 , srv638118987:84 , srv667501064:84 , srv685895590:84 , srv693234450:84 , srv823812947:84 , srv935954853:84 , srv1015443873:85 , srv1025042129:85 , srv1060755587:85 , srv1117232329:85 , srv1155923280:85 , srv1242927135:85 , srv1249242657:85 , srv1263533770:85 , srv126392810:85 , srv1317373061:85 , srv1468396867:85 , srv1509199719:85 , srv1512604359:85 , srv1560263249:85 , srv1596371857:85 , srv164418949:85 , srv1729945837:85 , srv1754317133:85 , srv1764106316:85 , srv177382926:85 , srv1814574887:85 , srv1822670328:85 , srv1833146081:85 , srv18684113:85 , srv1904097318:85 , srv1916962882:85 , srv1943112758:85 , srv2085926014:85 , srv2111619928:85 , srv2116271781:85 , srv212705111:85 , srv366590463:85 , srv517905361:85 , srv633919242:85 , srv705498543:85 , srv711491679:85 , srv741553599:85 , srv779845732:85 , srv82121537:85 , srv860829113:85 , srv863713138:85 , srv901350315:85 , srv912914686:85 , srv94857398:85 , srv996068448:85 , srv1404390838:86 , srv1520435814:86 , srv1717750042:86 , srv286714588:86 , srv355704197:86 , srv638523012:86 , srv15162245:638 }
2023-07-13 14:53:13,500 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=365 (was 365), ProcessCount=172 (was 172), AvailableMemoryMB=7890 (was 8730)
2023-07-13 14:53:13,510 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=365, ProcessCount=172, AvailableMemoryMB=7890
2023-07-13 14:53:13,510 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-07-13 14:53:13,511 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-07-13 14:53:13,525 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1071836772:84 , srv1143120337:84 , srv1215630462:84 , srv1307650235:84 , srv1485269043:84 , srv1512362835:84 , srv161184700:84 , srv1644400427:84 , srv1686637434:84 , srv171123754:84 , srv171205501:84 , srv1806786765:84 , srv1970884507:84 , srv2008082607:84 , srv2085080568:84 , srv223922185:84 , srv227902771:84 , srv274063871:84 , srv291696304:84 , srv325490213:84 , srv385343311:84 , srv418050244:84 , srv476577186:84 , srv529729204:84 , srv576669010:84 , srv734007332:84 , srv757118586:84 , srv843888010:84 , srv895041694:84 , srv1753742363:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-07-13 14:53:13,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-13 14:53:13,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv529729204=24, srv1512362835=5, srv291696304=19, srv734007332=26, srv1143120337=1, srv274063871=18, srv171205501=10, srv161184700=6, srv418050244=22, srv2085080568=15, srv1686637434=8, srv1485269043=4, srv171123754=9, srv227902771=17, srv476577186=23, srv223922185=16, srv895041694=29, srv1215630462=2, srv576669010=25, srv843888010=28, srv1071836772=0, srv2008082607=14, srv1307650235=3, srv757118586=27, srv325490213=20, srv385343311=21, srv1970884507=13, srv1644400427=7, srv1753742363=11, srv1806786765=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-13 14:53:13,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-13 14:53:13,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-13 14:53:13,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-13 14:53:13,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-13 14:53:13,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-13 14:53:13,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-13 14:53:13,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-13 14:53:13,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-13 14:53:13,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-13 14:53:13,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-13 14:53:13,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-13 14:53:13,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-13 14:53:13,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-13 14:53:13,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-13 14:53:13,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-13 14:53:13,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-13 14:53:13,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-13 14:53:13,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-13 14:53:13,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-13 14:53:13,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-13 14:53:13,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-13 14:53:13,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-13 14:53:13,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-13 14:53:13,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-13 14:53:13,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-13 14:53:13,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-13 14:53:13,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-13 14:53:13,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-13 14:53:13,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-13 14:53:13,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-13 14:53:13,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-07-13 14:53:13,570 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0983328291997498, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-13 14:53:13,570 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912502886798227, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0983328291997498, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-07-13 14:55:13,548 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5416212 different iterations.  Found a solution that moves 1852 regions; Going from a computed imbalance of 0.9912502886798227 to a new imbalance of 4.322660390073086E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6859259259259259, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-13 14:55:13,556 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1071836772:90 , srv1143120337:90 , srv1215630462:90 , srv1307650235:90 , srv1485269043:90 , srv1512362835:90 , srv161184700:90 , srv1644400427:90 , srv1686637434:90 , srv171123754:90 , srv171205501:90 , srv1753742363:90 , srv1806786765:90 , srv1970884507:90 , srv2008082607:90 , srv2085080568:90 , srv223922185:90 , srv227902771:90 , srv274063871:90 , srv291696304:90 , srv325490213:90 , srv385343311:90 , srv418050244:90 , srv476577186:90 , srv529729204:90 , srv576669010:90 , srv734007332:90 , srv757118586:90 , srv843888010:90 , srv895041694:90 }
2023-07-13 14:55:13,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-07-13 14:55:13,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv529729204=24, srv1512362835=5, srv291696304=19, srv734007332=26, srv1143120337=1, srv274063871=18, srv171205501=10, srv161184700=6, srv418050244=22, srv2085080568=15, srv1686637434=8, srv1485269043=4, srv171123754=9, srv227902771=17, srv476577186=23, srv223922185=16, srv895041694=29, srv1215630462=2, srv576669010=25, srv843888010=28, srv1071836772=0, srv2008082607=14, srv1307650235=3, srv757118586=27, srv325490213=20, srv385343311=21, srv1970884507=13, srv1644400427=7, srv1753742363=11, srv1806786765=12} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-07-13 14:55:13,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-07-13 14:55:13,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-07-13 14:55:13,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-07-13 14:55:13,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-07-13 14:55:13,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-07-13 14:55:13,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-07-13 14:55:13,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-07-13 14:55:13,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-07-13 14:55:13,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-07-13 14:55:13,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-07-13 14:55:13,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-07-13 14:55:13,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-07-13 14:55:13,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-07-13 14:55:13,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-07-13 14:55:13,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-07-13 14:55:13,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-07-13 14:55:13,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-07-13 14:55:13,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-07-13 14:55:13,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-07-13 14:55:13,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-07-13 14:55:13,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-07-13 14:55:13,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-07-13 14:55:13,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-07-13 14:55:13,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-07-13 14:55:13,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-07-13 14:55:13,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-07-13 14:55:13,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-07-13 14:55:13,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-07-13 14:55:13,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-07-13 14:55:13,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-07-13 14:55:13,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-07-13 14:55:13,578 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-07-13 14:55:13,579 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1071836772:90 , srv1143120337:90 , srv1215630462:90 , srv1307650235:90 , srv1485269043:90 , srv1512362835:90 , srv161184700:90 , srv1644400427:90 , srv1686637434:90 , srv171123754:90 , srv171205501:90 , srv1753742363:90 , srv1806786765:90 , srv1970884507:90 , srv2008082607:90 , srv2085080568:90 , srv223922185:90 , srv227902771:90 , srv274063871:90 , srv291696304:90 , srv325490213:90 , srv385343311:90 , srv418050244:90 , srv476577186:90 , srv529729204:90 , srv576669010:90 , srv734007332:90 , srv757118586:90 , srv843888010:90 , srv895041694:90 }
2023-07-13 14:55:13,591 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=220 (was 365), ProcessCount=170 (was 172), AvailableMemoryMB=7956 (was 7890) - AvailableMemoryMB LEAK? -