2023-11-23 01:58:31,033 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-11-23 01:58:31,373 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-11-23 01:58:31,485 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-11-23 01:58:31,553 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-23 01:58:31,561 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-23 01:58:31,574 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=488, ProcessCount=169, AvailableMemoryMB=13308
2023-11-23 01:58:31,576 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-23 01:58:31,577 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-23 01:58:31,718 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1016042292:84 , srv1037780728:84 , srv1064914756:84 , srv1078946624:84 , srv1120534152:84 , srv1122593515:84 , srv1161201484:84 , srv1171723540:84 , srv1204114145:84 , srv1238253673:84 , srv1267584770:84 , srv1282214521:84 , srv1334394762:84 , srv133491636:84 , srv1363626108:84 , srv1380367071:84 , srv1384702582:84 , srv1408771750:84 , srv1421132009:84 , srv1426233255:84 , srv1428499488:84 , srv1437460413:84 , srv1509699090:84 , srv1518006394:84 , srv1524220810:84 , srv1560404237:84 , srv1604151659:84 , srv1632242373:84 , srv1683467308:84 , srv1691841015:84 , srv1703831557:84 , srv1707473424:84 , srv1712089142:84 , srv1728222862:84 , srv1747782612:84 , srv1761542066:84 , srv1773144917:84 , srv1832566714:84 , srv1847852533:84 , srv1852248465:84 , srv1871580138:84 , srv1902863925:84 , srv1929676688:84 , srv1950112136:84 , srv1974620710:84 , srv1974756577:84 , srv1980389129:84 , srv2026578456:84 , srv2040277732:84 , srv2044029384:84 , srv2065123265:84 , srv2071432451:84 , srv2103470022:84 , srv2117837303:84 , srv2128189028:84 , srv2130205288:84 , srv2139733538:84 , srv252526444:84 , srv270381081:84 , srv272884270:84 , srv273136805:84 , srv284108103:84 , srv285628987:84 , srv295673423:84 , srv34208514:84 , srv361254433:84 , srv366339969:84 , srv393210828:84 , srv436476017:84 , srv524901015:84 , srv529286076:84 , srv547410641:84 , srv579517430:84 , srv582285583:84 , srv582890445:84 , srv622221498:84 , srv644091570:84 , srv658218779:84 , srv665702047:84 , srv679678129:84 , srv685610725:84 , srv699872062:84 , srv72104483:84 , srv734389475:84 , srv74821594:84 , srv758512412:84 , srv771987511:84 , srv772570290:84 , srv799775277:84 , srv820189990:84 , srv820445645:84 , srv820581874:84 , srv82631449:84 , srv84550751:84 , srv847749930:84 , srv872637492:84 , srv878034450:84 , srv894010756:84 , srv978095466:84 , srv1695917394:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-11-23 01:58:31,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-23 01:58:31,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv72104483=83, srv84550751=94, srv547410641=72, srv1524220810=24, srv1204114145=8, srv436476017=69, srv820445645=91, srv1712089142=33, srv1437460413=21, srv34208514=65, srv582285583=74, srv679678129=80, srv285628987=63, srv2040277732=49, srv1707473424=32, srv1428499488=20, srv1078946624=3, srv1334394762=12, srv622221498=76, srv1509699090=22, srv1847852533=39, srv582890445=75, srv295673423=64, srv1282214521=11, srv1037780728=1, srv1747782612=35, srv1695917394=30, srv699872062=82, srv361254433=66, srv1380367071=15, srv366339969=67, srv1728222862=34, srv847749930=95, srv1161201484=6, srv2130205288=56, srv284108103=62, srv1974620710=45, srv2026578456=48, srv1929676688=43, srv2044029384=50, srv1632242373=27, srv1518006394=23, srv1950112136=44, srv1238253673=9, srv1691841015=29, srv1408771750=17, srv1122593515=5, srv658218779=78, srv758512412=86, srv252526444=58, srv1761542066=36, srv1852248465=40, srv872637492=96, srv1902863925=42, srv894010756=98, srv1604151659=26, srv1871580138=41, srv2128189028=55, srv1267584770=10, srv1363626108=14, srv272884270=60, srv820581874=92, srv1384702582=16, srv1426233255=19, srv1064914756=2, srv1974756577=46, srv1120534152=4, srv133491636=13, srv2071432451=52, srv771987511=87, srv524901015=70, srv665702047=79, srv579517430=73, srv1421132009=18, srv2117837303=54, srv1773144917=37, srv1683467308=28, srv644091570=77, srv2103470022=53, srv273136805=61, srv1171723540=7, srv1980389129=47, srv74821594=85, srv2139733538=57, srv1560404237=25, srv1703831557=31, srv799775277=89, srv1016042292=0, srv734389475=84, srv82631449=93, srv820189990=90, srv2065123265=51, srv878034450=97, srv685610725=81, srv270381081=59, srv529286076=71, srv772570290=88, srv978095466=99, srv1832566714=38, srv393210828=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-23 01:58:31,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-23 01:58:31,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-23 01:58:31,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-23 01:58:31,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-23 01:58:31,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-23 01:58:31,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-23 01:58:31,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-23 01:58:31,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-23 01:58:31,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-23 01:58:31,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-23 01:58:31,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-23 01:58:31,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-23 01:58:31,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-23 01:58:31,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-23 01:58:31,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-23 01:58:31,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-23 01:58:31,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-23 01:58:31,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-23 01:58:31,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-23 01:58:31,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-23 01:58:31,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-23 01:58:31,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-23 01:58:31,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-23 01:58:31,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-23 01:58:31,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-23 01:58:31,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-23 01:58:31,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-23 01:58:31,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-23 01:58:31,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-23 01:58:31,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-23 01:58:31,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-23 01:58:31,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-23 01:58:31,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-23 01:58:31,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-23 01:58:31,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-23 01:58:31,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-23 01:58:31,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-23 01:58:31,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-23 01:58:31,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-23 01:58:31,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-23 01:58:31,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-23 01:58:31,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-23 01:58:31,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-23 01:58:31,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-23 01:58:31,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-23 01:58:31,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-23 01:58:31,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-23 01:58:31,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-23 01:58:31,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-23 01:58:31,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-23 01:58:31,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-23 01:58:31,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-23 01:58:31,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-23 01:58:31,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-23 01:58:31,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-23 01:58:31,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-23 01:58:31,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-23 01:58:31,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-23 01:58:31,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-23 01:58:31,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-23 01:58:31,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-23 01:58:31,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-23 01:58:31,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-23 01:58:31,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-23 01:58:31,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-23 01:58:31,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-23 01:58:31,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-23 01:58:31,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-23 01:58:31,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-23 01:58:31,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-23 01:58:31,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-23 01:58:31,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-23 01:58:31,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-23 01:58:31,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-23 01:58:31,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-23 01:58:31,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-23 01:58:31,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-23 01:58:31,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-23 01:58:31,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-23 01:58:31,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-23 01:58:31,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-23 01:58:31,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-23 01:58:31,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-23 01:58:31,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-23 01:58:31,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-23 01:58:31,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-23 01:58:31,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-23 01:58:31,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-23 01:58:31,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-23 01:58:31,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-23 01:58:31,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-23 01:58:31,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-23 01:58:31,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-23 01:58:31,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-23 01:58:31,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-23 01:58:31,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-23 01:58:31,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-23 01:58:31,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-23 01:58:31,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-23 01:58:31,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-23 01:58:31,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-23 01:58:32,190 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-23 01:58:32,191 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-23 01:58:32,191 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-23 01:58:32,541 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 678 ms to try 5000 different iterations.  Found a solution that moves 4645 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1857212372852601. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040719002464953205, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05625419300995169, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5161111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040719002464953205, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18091666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24841666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-23 01:58:32,555 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2139733538:81 , srv644091570:81 , srv1122593515:82 , srv1064914756:83 , srv1238253673:83 , srv1380367071:83 , srv1408771750:83 , srv1437460413:83 , srv1852248465:83 , srv1871580138:83 , srv1974620710:83 , srv361254433:83 , srv579517430:83 , srv74821594:83 , srv1509699090:84 , srv1604151659:84 , srv1683467308:84 , srv1847852533:84 , srv252526444:84 , srv270381081:84 , srv273136805:84 , srv393210828:84 , srv771987511:84 , srv872637492:84 , srv1428499488:85 , srv1703831557:85 , srv1761542066:85 , srv2040277732:85 , srv34208514:85 , srv529286076:85 , srv665702047:85 , srv699872062:85 , srv772570290:85 , srv847749930:85 , srv878034450:85 , srv1171723540:86 , srv1363626108:86 , srv1384702582:86 , srv1518006394:86 , srv1524220810:86 , srv1728222862:86 , srv1902863925:86 , srv2117837303:86 , srv295673423:86 , srv547410641:86 , srv582890445:86 , srv734389475:86 , srv820189990:86 , srv820445645:86 , srv820581874:86 , srv1016042292:87 , srv1078946624:87 , srv1204114145:87 , srv1426233255:87 , srv1560404237:87 , srv1632242373:87 , srv1707473424:87 , srv1929676688:87 , srv2103470022:87 , srv2128189028:87 , srv272884270:87 , srv582285583:87 , srv622221498:87 , srv72104483:87 , srv799775277:87 , srv82631449:87 , srv84550751:87 , srv978095466:87 , srv1037780728:88 , srv1161201484:88 , srv1334394762:88 , srv1691841015:88 , srv1712089142:88 , srv1773144917:88 , srv1832566714:88 , srv2044029384:88 , srv2065123265:88 , srv284108103:88 , srv285628987:88 , srv366339969:88 , srv436476017:88 , srv524901015:88 , srv685610725:88 , srv1120534152:89 , srv1267584770:89 , srv1282214521:89 , srv133491636:89 , srv1747782612:89 , srv1974756577:89 , srv1980389129:89 , srv2026578456:89 , srv658218779:89 , srv679678129:89 , srv2071432451:90 , srv758512412:90 , srv1950112136:91 , srv894010756:91 , srv1421132009:92 , srv2130205288:96 , srv1695917394:452 }
2023-11-23 01:58:32,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-23 01:58:32,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv72104483=83, srv84550751=94, srv547410641=72, srv1524220810=24, srv1204114145=8, srv436476017=69, srv820445645=91, srv1712089142=33, srv1437460413=21, srv34208514=65, srv582285583=74, srv679678129=80, srv285628987=63, srv2040277732=49, srv1707473424=32, srv1428499488=20, srv1078946624=3, srv1334394762=12, srv622221498=76, srv1509699090=22, srv1847852533=39, srv582890445=75, srv295673423=64, srv1282214521=11, srv1037780728=1, srv1747782612=35, srv1695917394=30, srv699872062=82, srv361254433=66, srv1380367071=15, srv366339969=67, srv1728222862=34, srv847749930=95, srv1161201484=6, srv2130205288=56, srv284108103=62, srv1974620710=45, srv2026578456=48, srv1929676688=43, srv2044029384=50, srv1632242373=27, srv1518006394=23, srv1950112136=44, srv1238253673=9, srv1691841015=29, srv1408771750=17, srv1122593515=5, srv658218779=78, srv758512412=86, srv252526444=58, srv1761542066=36, srv1852248465=40, srv872637492=96, srv1902863925=42, srv894010756=98, srv1604151659=26, srv1871580138=41, srv2128189028=55, srv1267584770=10, srv1363626108=14, srv272884270=60, srv820581874=92, srv1384702582=16, srv1426233255=19, srv1064914756=2, srv1974756577=46, srv1120534152=4, srv133491636=13, srv2071432451=52, srv771987511=87, srv524901015=70, srv665702047=79, srv579517430=73, srv1421132009=18, srv2117837303=54, srv1773144917=37, srv1683467308=28, srv644091570=77, srv2103470022=53, srv273136805=61, srv1171723540=7, srv1980389129=47, srv74821594=85, srv2139733538=57, srv1560404237=25, srv1703831557=31, srv799775277=89, srv1016042292=0, srv734389475=84, srv82631449=93, srv820189990=90, srv2065123265=51, srv878034450=97, srv685610725=81, srv270381081=59, srv529286076=71, srv772570290=88, srv978095466=99, srv1832566714=38, srv393210828=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-23 01:58:32,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-23 01:58:32,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-23 01:58:32,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-23 01:58:32,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-23 01:58:32,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-23 01:58:32,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-23 01:58:32,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-23 01:58:32,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-23 01:58:32,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-23 01:58:32,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-23 01:58:32,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-23 01:58:32,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-23 01:58:32,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-23 01:58:32,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-23 01:58:32,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-23 01:58:32,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-23 01:58:32,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-23 01:58:32,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-23 01:58:32,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-23 01:58:32,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-23 01:58:32,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-23 01:58:32,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-23 01:58:32,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-23 01:58:32,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-23 01:58:32,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-23 01:58:32,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-23 01:58:32,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-23 01:58:32,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-23 01:58:32,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-23 01:58:32,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-23 01:58:32,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-23 01:58:32,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-23 01:58:32,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-23 01:58:32,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-23 01:58:32,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-23 01:58:32,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-23 01:58:32,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-23 01:58:32,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-23 01:58:32,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-23 01:58:32,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-23 01:58:32,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-23 01:58:32,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-23 01:58:32,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-23 01:58:32,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-23 01:58:32,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-23 01:58:32,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-23 01:58:32,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-23 01:58:32,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-23 01:58:32,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-23 01:58:32,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-23 01:58:32,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-23 01:58:32,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-23 01:58:32,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-23 01:58:32,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-23 01:58:32,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-23 01:58:32,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-23 01:58:32,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-23 01:58:32,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-23 01:58:32,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-23 01:58:32,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-23 01:58:32,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-23 01:58:32,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-23 01:58:32,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-23 01:58:32,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-23 01:58:32,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-23 01:58:32,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-23 01:58:32,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-23 01:58:32,592 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-23 01:58:32,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-23 01:58:32,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-23 01:58:32,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-23 01:58:32,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-23 01:58:32,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-23 01:58:32,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-23 01:58:32,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-23 01:58:32,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-23 01:58:32,593 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-23 01:58:32,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-23 01:58:32,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-23 01:58:32,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-23 01:58:32,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-23 01:58:32,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-23 01:58:32,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-23 01:58:32,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-23 01:58:32,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-23 01:58:32,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-23 01:58:32,594 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-23 01:58:32,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-23 01:58:32,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-23 01:58:32,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-23 01:58:32,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-23 01:58:32,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-23 01:58:32,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-23 01:58:32,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-23 01:58:32,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-23 01:58:32,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-23 01:58:32,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-23 01:58:32,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-23 01:58:32,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-23 01:58:32,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-23 01:58:32,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-23 01:58:32,729 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040719002464953205, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05625419300995169, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040719002464953205, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18091666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24841666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-23 01:58:32,730 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-23 01:58:32,730 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18568871230008968, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040719002464953205, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05625419300995169, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040719002464953205, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18091666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24841666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-23 01:58:32,924 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 299 ms to try 5000 different iterations.  Found a solution that moves 2664 regions; Going from a computed imbalance of 0.18568871230008968 to a new imbalance of 0.016541132142762573. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023736824503520083); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038469601157843404, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.296, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023736824503520083); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0135); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04533333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-23 01:58:32,935 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2130205288:76 , srv1421132009:80 , srv1950112136:80 , srv133491636:81 , srv1747782612:81 , srv679678129:81 , srv894010756:81 , srv1120534152:82 , srv1929676688:82 , srv2026578456:82 , srv2071432451:82 , srv366339969:82 , srv758512412:82 , srv1037780728:83 , srv1267584770:83 , srv1282214521:83 , srv1712089142:83 , srv1832566714:83 , srv1974756577:83 , srv658218779:83 , srv1078946624:84 , srv1161201484:84 , srv1363626108:84 , srv1426233255:84 , srv1773144917:84 , srv1980389129:84 , srv272884270:84 , srv284108103:84 , srv285628987:84 , srv436476017:84 , srv799775277:84 , srv1016042292:85 , srv1334394762:85 , srv1524220810:85 , srv1560404237:85 , srv1632242373:85 , srv1691841015:85 , srv1703831557:85 , srv2044029384:85 , srv2103470022:85 , srv295673423:85 , srv34208514:85 , srv622221498:85 , srv685610725:85 , srv820189990:85 , srv82631449:85 , srv978095466:85 , srv1384702582:86 , srv1902863925:86 , srv2040277732:86 , srv2117837303:86 , srv2128189028:86 , srv524901015:86 , srv529286076:86 , srv582285583:86 , srv582890445:86 , srv699872062:86 , srv820445645:86 , srv820581874:86 , srv84550751:86 , srv847749930:86 , srv1064914756:87 , srv1171723540:87 , srv1509699090:87 , srv1604151659:87 , srv1707473424:87 , srv1728222862:87 , srv1871580138:87 , srv2065123265:87 , srv252526444:87 , srv273136805:87 , srv393210828:87 , srv547410641:87 , srv579517430:87 , srv665702047:87 , srv72104483:87 , srv771987511:87 , srv772570290:87 , srv872637492:87 , srv1380367071:88 , srv1408771750:88 , srv1437460413:88 , srv1518006394:88 , srv1683467308:88 , srv1761542066:88 , srv1847852533:88 , srv270381081:88 , srv878034450:88 , srv1122593515:89 , srv1204114145:89 , srv1238253673:89 , srv1852248465:89 , srv1974620710:89 , srv361254433:89 , srv644091570:89 , srv1428499488:90 , srv2139733538:90 , srv74821594:91 , srv734389475:94 , srv1695917394:533 }
2023-11-23 01:58:32,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-23 01:58:32,941 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv72104483=83, srv84550751=94, srv547410641=72, srv1524220810=24, srv1204114145=8, srv436476017=69, srv820445645=91, srv1712089142=33, srv1437460413=21, srv34208514=65, srv582285583=74, srv679678129=80, srv285628987=63, srv2040277732=49, srv1707473424=32, srv1428499488=20, srv1078946624=3, srv1334394762=12, srv622221498=76, srv1509699090=22, srv1847852533=39, srv582890445=75, srv295673423=64, srv1282214521=11, srv1037780728=1, srv1747782612=35, srv1695917394=30, srv699872062=82, srv361254433=66, srv1380367071=15, srv366339969=67, srv1728222862=34, srv847749930=95, srv1161201484=6, srv2130205288=56, srv284108103=62, srv1974620710=45, srv2026578456=48, srv1929676688=43, srv2044029384=50, srv1632242373=27, srv1518006394=23, srv1950112136=44, srv1238253673=9, srv1691841015=29, srv1408771750=17, srv1122593515=5, srv658218779=78, srv758512412=86, srv252526444=58, srv1761542066=36, srv1852248465=40, srv872637492=96, srv1902863925=42, srv894010756=98, srv1604151659=26, srv1871580138=41, srv2128189028=55, srv1267584770=10, srv1363626108=14, srv272884270=60, srv820581874=92, srv1384702582=16, srv1426233255=19, srv1064914756=2, srv1974756577=46, srv1120534152=4, srv133491636=13, srv2071432451=52, srv771987511=87, srv524901015=70, srv665702047=79, srv579517430=73, srv1421132009=18, srv2117837303=54, srv1773144917=37, srv1683467308=28, srv644091570=77, srv2103470022=53, srv273136805=61, srv1171723540=7, srv1980389129=47, srv74821594=85, srv2139733538=57, srv1560404237=25, srv1703831557=31, srv799775277=89, srv1016042292=0, srv734389475=84, srv82631449=93, srv820189990=90, srv2065123265=51, srv878034450=97, srv685610725=81, srv270381081=59, srv529286076=71, srv772570290=88, srv978095466=99, srv1832566714=38, srv393210828=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-23 01:58:32,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-23 01:58:32,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-23 01:58:32,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-23 01:58:32,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-23 01:58:32,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-23 01:58:32,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-23 01:58:32,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-23 01:58:32,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-23 01:58:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-23 01:58:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-23 01:58:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-23 01:58:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-23 01:58:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-23 01:58:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-23 01:58:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-23 01:58:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-23 01:58:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-23 01:58:32,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-23 01:58:32,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-23 01:58:32,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-23 01:58:32,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-23 01:58:32,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-23 01:58:32,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-23 01:58:32,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-23 01:58:32,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-23 01:58:32,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-23 01:58:32,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-23 01:58:32,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-23 01:58:32,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-23 01:58:32,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-23 01:58:32,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-23 01:58:32,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-23 01:58:32,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-23 01:58:32,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-23 01:58:32,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-23 01:58:32,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-23 01:58:32,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-23 01:58:32,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-23 01:58:32,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-23 01:58:32,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-23 01:58:32,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-23 01:58:32,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-23 01:58:32,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-23 01:58:32,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-23 01:58:32,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-23 01:58:32,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-23 01:58:32,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-23 01:58:32,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-23 01:58:32,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-23 01:58:32,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-23 01:58:32,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-23 01:58:32,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-23 01:58:32,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-23 01:58:32,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-23 01:58:32,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-23 01:58:32,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-23 01:58:32,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-23 01:58:32,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-23 01:58:32,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-23 01:58:32,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-23 01:58:32,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-23 01:58:32,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-23 01:58:32,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-23 01:58:32,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-23 01:58:32,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-23 01:58:32,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-23 01:58:32,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-23 01:58:32,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-23 01:58:32,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-23 01:58:32,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-23 01:58:32,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-23 01:58:32,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-23 01:58:32,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-23 01:58:32,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-23 01:58:32,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-23 01:58:32,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-23 01:58:32,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-23 01:58:32,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-23 01:58:32,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-23 01:58:32,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-23 01:58:32,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-23 01:58:32,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-23 01:58:32,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-23 01:58:32,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-23 01:58:32,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-23 01:58:32,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-23 01:58:32,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-23 01:58:32,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-23 01:58:32,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-23 01:58:32,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-23 01:58:32,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-23 01:58:32,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-23 01:58:32,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-23 01:58:32,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-23 01:58:32,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-23 01:58:32,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-23 01:58:32,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-23 01:58:32,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-23 01:58:32,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-23 01:58:32,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-23 01:58:32,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-23 01:58:33,100 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023736824503520083); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038469601157843404, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023736824503520083); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0135); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04533333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-23 01:58:33,100 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-23 01:58:33,101 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01652247841606848, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023736824503520083); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038469601157843404, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023736824503520083); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0135); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04533333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-23 01:58:33,338 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 339 ms to try 5000 different iterations.  Found a solution that moves 981 regions; Going from a computed imbalance of 0.01652247841606848 to a new imbalance of 8.09205952692963E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008571784860676212); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.014070529413628968); MoveCostFunction : (multiplier=7.0, imbalance=0.109, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008571784860676212); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-23 01:58:33,340 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv734389475:77 , srv1204114145:81 , srv1428499488:82 , srv1691841015:83 , srv1707473424:83 , srv2065123265:83 , srv2128189028:83 , srv72104483:83 , srv74821594:83 , srv1161201484:84 , srv1334394762:84 , srv1518006394:84 , srv1683467308:84 , srv1761542066:84 , srv1980389129:84 , srv2044029384:84 , srv2130205288:84 , srv284108103:84 , srv524901015:84 , srv547410641:84 , srv582285583:84 , srv622221498:84 , srv665702047:84 , srv772570290:84 , srv820581874:84 , srv84550751:84 , srv878034450:84 , srv1016042292:85 , srv1120534152:85 , srv1171723540:85 , srv1267584770:85 , srv1282214521:85 , srv1384702582:85 , srv1421132009:85 , srv1426233255:85 , srv1632242373:85 , srv1728222862:85 , srv1773144917:85 , srv1847852533:85 , srv1852248465:85 , srv1902863925:85 , srv2071432451:85 , srv2117837303:85 , srv270381081:85 , srv285628987:85 , srv361254433:85 , srv436476017:85 , srv685610725:85 , srv699872062:85 , srv820445645:85 , srv1078946624:86 , srv1380367071:86 , srv1509699090:86 , srv1524220810:86 , srv1560404237:86 , srv1703831557:86 , srv1712089142:86 , srv1832566714:86 , srv1950112136:86 , srv1974620710:86 , srv1974756577:86 , srv272884270:86 , srv393210828:86 , srv529286076:86 , srv582890445:86 , srv658218779:86 , srv758512412:86 , srv820189990:86 , srv82631449:86 , srv847749930:86 , srv978095466:86 , srv1037780728:87 , srv1064914756:87 , srv1122593515:87 , srv1238253673:87 , srv133491636:87 , srv1363626108:87 , srv1408771750:87 , srv1437460413:87 , srv1604151659:87 , srv1747782612:87 , srv1871580138:87 , srv2026578456:87 , srv2040277732:87 , srv2103470022:87 , srv2139733538:87 , srv252526444:87 , srv273136805:87 , srv295673423:87 , srv34208514:87 , srv366339969:87 , srv644091570:87 , srv679678129:87 , srv771987511:87 , srv799775277:87 , srv872637492:87 , srv1929676688:88 , srv894010756:88 , srv579517430:89 , srv1695917394:549 }
2023-11-23 01:58:33,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-23 01:58:33,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv72104483=83, srv84550751=94, srv547410641=72, srv1524220810=24, srv1204114145=8, srv436476017=69, srv820445645=91, srv1712089142=33, srv1437460413=21, srv34208514=65, srv582285583=74, srv679678129=80, srv285628987=63, srv2040277732=49, srv1707473424=32, srv1428499488=20, srv1078946624=3, srv1334394762=12, srv622221498=76, srv1509699090=22, srv1847852533=39, srv582890445=75, srv295673423=64, srv1282214521=11, srv1037780728=1, srv1747782612=35, srv1695917394=30, srv699872062=82, srv361254433=66, srv1380367071=15, srv366339969=67, srv1728222862=34, srv847749930=95, srv1161201484=6, srv2130205288=56, srv284108103=62, srv1974620710=45, srv2026578456=48, srv1929676688=43, srv2044029384=50, srv1632242373=27, srv1518006394=23, srv1950112136=44, srv1238253673=9, srv1691841015=29, srv1408771750=17, srv1122593515=5, srv658218779=78, srv758512412=86, srv252526444=58, srv1761542066=36, srv1852248465=40, srv872637492=96, srv1902863925=42, srv894010756=98, srv1604151659=26, srv1871580138=41, srv2128189028=55, srv1267584770=10, srv1363626108=14, srv272884270=60, srv820581874=92, srv1384702582=16, srv1426233255=19, srv1064914756=2, srv1974756577=46, srv1120534152=4, srv133491636=13, srv2071432451=52, srv771987511=87, srv524901015=70, srv665702047=79, srv579517430=73, srv1421132009=18, srv2117837303=54, srv1773144917=37, srv1683467308=28, srv644091570=77, srv2103470022=53, srv273136805=61, srv1171723540=7, srv1980389129=47, srv74821594=85, srv2139733538=57, srv1560404237=25, srv1703831557=31, srv799775277=89, srv1016042292=0, srv734389475=84, srv82631449=93, srv820189990=90, srv2065123265=51, srv878034450=97, srv685610725=81, srv270381081=59, srv529286076=71, srv772570290=88, srv978095466=99, srv1832566714=38, srv393210828=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-23 01:58:33,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-23 01:58:33,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-23 01:58:33,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-23 01:58:33,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-23 01:58:33,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-23 01:58:33,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-23 01:58:33,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-23 01:58:33,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-23 01:58:33,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-23 01:58:33,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-23 01:58:33,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-23 01:58:33,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-23 01:58:33,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-23 01:58:33,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-23 01:58:33,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-23 01:58:33,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-23 01:58:33,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-23 01:58:33,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-23 01:58:33,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-23 01:58:33,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-23 01:58:33,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-23 01:58:33,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-23 01:58:33,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-23 01:58:33,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-23 01:58:33,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-23 01:58:33,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-23 01:58:33,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-23 01:58:33,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-23 01:58:33,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-23 01:58:33,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-23 01:58:33,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-23 01:58:33,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-23 01:58:33,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-23 01:58:33,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-23 01:58:33,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-23 01:58:33,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-23 01:58:33,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-23 01:58:33,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-23 01:58:33,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-23 01:58:33,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-23 01:58:33,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-23 01:58:33,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-23 01:58:33,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-23 01:58:33,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-23 01:58:33,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-23 01:58:33,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-23 01:58:33,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-23 01:58:33,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-23 01:58:33,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-23 01:58:33,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-23 01:58:33,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-23 01:58:33,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-23 01:58:33,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-23 01:58:33,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-23 01:58:33,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-23 01:58:33,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-23 01:58:33,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-23 01:58:33,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-23 01:58:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-23 01:58:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-23 01:58:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-23 01:58:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-23 01:58:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-23 01:58:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-23 01:58:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-23 01:58:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-23 01:58:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-23 01:58:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-23 01:58:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-23 01:58:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-23 01:58:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-23 01:58:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-23 01:58:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-23 01:58:33,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-23 01:58:33,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-23 01:58:33,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-23 01:58:33,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-23 01:58:33,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-23 01:58:33,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-23 01:58:33,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-23 01:58:33,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-23 01:58:33,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-23 01:58:33,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-23 01:58:33,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-23 01:58:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-23 01:58:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-23 01:58:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-23 01:58:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-23 01:58:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-23 01:58:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-23 01:58:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-23 01:58:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-23 01:58:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-23 01:58:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-23 01:58:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-23 01:58:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-23 01:58:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-23 01:58:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-23 01:58:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-23 01:58:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-23 01:58:33,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-23 01:58:33,469 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008571784860676212); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.014070529413628968); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008571784860676212); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-23 01:58:33,469 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-23 01:58:33,469 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.023368438765566E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008571784860676212); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.014070529413628968); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008571784860676212); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-23 01:58:33,714 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 329 ms to try 5000 different iterations.  Found a solution that moves 318 regions; Going from a computed imbalance of 8.023368438765566E-4 to a new imbalance of 6.06224368680293E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00259499648053841); MoveCostFunction : (multiplier=7.0, imbalance=0.035333333333333335, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-23 01:58:33,715 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv894010756:82 , srv2103470022:83 , srv579517430:83 , srv1037780728:84 , srv1078946624:84 , srv1122593515:84 , srv1171723540:84 , srv1238253673:84 , srv1384702582:84 , srv1408771750:84 , srv1437460413:84 , srv1518006394:84 , srv1560404237:84 , srv1604151659:84 , srv1728222862:84 , srv1832566714:84 , srv1902863925:84 , srv1974620710:84 , srv1974756577:84 , srv2026578456:84 , srv2040277732:84 , srv2065123265:84 , srv2139733538:84 , srv252526444:84 , srv295673423:84 , srv34208514:84 , srv524901015:84 , srv582890445:84 , srv658218779:84 , srv685610725:84 , srv758512412:84 , srv771987511:84 , srv799775277:84 , srv82631449:84 , srv872637492:84 , srv978095466:84 , srv1016042292:85 , srv1064914756:85 , srv1204114145:85 , srv1267584770:85 , srv1282214521:85 , srv1334394762:85 , srv133491636:85 , srv1363626108:85 , srv1380367071:85 , srv1421132009:85 , srv1428499488:85 , srv1509699090:85 , srv1524220810:85 , srv1632242373:85 , srv1707473424:85 , srv1712089142:85 , srv1747782612:85 , srv1761542066:85 , srv1773144917:85 , srv1847852533:85 , srv1852248465:85 , srv1871580138:85 , srv1929676688:85 , srv1950112136:85 , srv1980389129:85 , srv2044029384:85 , srv2071432451:85 , srv2117837303:85 , srv270381081:85 , srv272884270:85 , srv273136805:85 , srv285628987:85 , srv361254433:85 , srv366339969:85 , srv393210828:85 , srv436476017:85 , srv529286076:85 , srv547410641:85 , srv582285583:85 , srv644091570:85 , srv679678129:85 , srv72104483:85 , srv734389475:85 , srv74821594:85 , srv820189990:85 , srv820445645:85 , srv84550751:85 , srv847749930:85 , srv878034450:85 , srv1161201484:86 , srv1426233255:86 , srv1691841015:86 , srv1703831557:86 , srv2128189028:86 , srv2130205288:86 , srv284108103:86 , srv622221498:86 , srv665702047:86 , srv699872062:86 , srv820581874:86 , srv1120534152:87 , srv1683467308:87 , srv772570290:87 , srv1695917394:608 }
2023-11-23 01:58:33,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-23 01:58:33,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv72104483=83, srv84550751=94, srv547410641=72, srv1524220810=24, srv1204114145=8, srv436476017=69, srv820445645=91, srv1712089142=33, srv1437460413=21, srv34208514=65, srv582285583=74, srv679678129=80, srv285628987=63, srv2040277732=49, srv1707473424=32, srv1428499488=20, srv1078946624=3, srv1334394762=12, srv622221498=76, srv1509699090=22, srv1847852533=39, srv582890445=75, srv295673423=64, srv1282214521=11, srv1037780728=1, srv1747782612=35, srv1695917394=30, srv699872062=82, srv361254433=66, srv1380367071=15, srv366339969=67, srv1728222862=34, srv847749930=95, srv1161201484=6, srv2130205288=56, srv284108103=62, srv1974620710=45, srv2026578456=48, srv1929676688=43, srv2044029384=50, srv1632242373=27, srv1518006394=23, srv1950112136=44, srv1238253673=9, srv1691841015=29, srv1408771750=17, srv1122593515=5, srv658218779=78, srv758512412=86, srv252526444=58, srv1761542066=36, srv1852248465=40, srv872637492=96, srv1902863925=42, srv894010756=98, srv1604151659=26, srv1871580138=41, srv2128189028=55, srv1267584770=10, srv1363626108=14, srv272884270=60, srv820581874=92, srv1384702582=16, srv1426233255=19, srv1064914756=2, srv1974756577=46, srv1120534152=4, srv133491636=13, srv2071432451=52, srv771987511=87, srv524901015=70, srv665702047=79, srv579517430=73, srv1421132009=18, srv2117837303=54, srv1773144917=37, srv1683467308=28, srv644091570=77, srv2103470022=53, srv273136805=61, srv1171723540=7, srv1980389129=47, srv74821594=85, srv2139733538=57, srv1560404237=25, srv1703831557=31, srv799775277=89, srv1016042292=0, srv734389475=84, srv82631449=93, srv820189990=90, srv2065123265=51, srv878034450=97, srv685610725=81, srv270381081=59, srv529286076=71, srv772570290=88, srv978095466=99, srv1832566714=38, srv393210828=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-23 01:58:33,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-23 01:58:33,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-23 01:58:33,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-23 01:58:33,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-23 01:58:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-23 01:58:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-23 01:58:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-23 01:58:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-23 01:58:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-23 01:58:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-23 01:58:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-23 01:58:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-23 01:58:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-23 01:58:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-23 01:58:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-23 01:58:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-23 01:58:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-23 01:58:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-23 01:58:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-23 01:58:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-23 01:58:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-23 01:58:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-23 01:58:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-23 01:58:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-23 01:58:33,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-23 01:58:33,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-23 01:58:33,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-23 01:58:33,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-23 01:58:33,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-23 01:58:33,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-23 01:58:33,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-23 01:58:33,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-23 01:58:33,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-23 01:58:33,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-23 01:58:33,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-23 01:58:33,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-23 01:58:33,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-23 01:58:33,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-23 01:58:33,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-23 01:58:33,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-23 01:58:33,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-23 01:58:33,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-23 01:58:33,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-23 01:58:33,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-23 01:58:33,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-23 01:58:33,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-23 01:58:33,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-23 01:58:33,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-23 01:58:33,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-23 01:58:33,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-23 01:58:33,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-23 01:58:33,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-23 01:58:33,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-23 01:58:33,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-23 01:58:33,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-23 01:58:33,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-23 01:58:33,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-23 01:58:33,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-23 01:58:33,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-23 01:58:33,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-23 01:58:33,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-23 01:58:33,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-23 01:58:33,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-23 01:58:33,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-23 01:58:33,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-23 01:58:33,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-23 01:58:33,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-23 01:58:33,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-23 01:58:33,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-23 01:58:33,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-23 01:58:33,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-23 01:58:33,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-23 01:58:33,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-23 01:58:33,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-23 01:58:33,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-23 01:58:33,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-23 01:58:33,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-23 01:58:33,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-23 01:58:33,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-23 01:58:33,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-23 01:58:33,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-23 01:58:33,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-23 01:58:33,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-23 01:58:33,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-23 01:58:33,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-23 01:58:33,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-23 01:58:33,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-23 01:58:33,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-23 01:58:33,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-23 01:58:33,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-23 01:58:33,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-23 01:58:33,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-23 01:58:33,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-23 01:58:33,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-23 01:58:33,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-23 01:58:33,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-23 01:58:33,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-23 01:58:33,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-23 01:58:33,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-23 01:58:33,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-23 01:58:33,743 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-23 01:58:33,820 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00259499648053841); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-23 01:58:33,821 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-23 01:58:33,821 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.83957532761666E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.5313429231318235E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00259499648053841); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.5313429231318235E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-23 01:58:33,970 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 221 ms to try 5000 different iterations.  Found a solution that moves 44 regions; Going from a computed imbalance of 5.83957532761666E-5 to a new imbalance of 9.101284824356401E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.004888888888888889); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-23 01:58:33,971 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1120534152:83 , srv1238253673:83 , srv1683467308:83 , srv273136805:83 , srv772570290:83 , srv1016042292:84 , srv1037780728:84 , srv1064914756:84 , srv1122593515:84 , srv1161201484:84 , srv1171723540:84 , srv1204114145:84 , srv1267584770:84 , srv1282214521:84 , srv1334394762:84 , srv133491636:84 , srv1363626108:84 , srv1380367071:84 , srv1408771750:84 , srv1421132009:84 , srv1426233255:84 , srv1428499488:84 , srv1437460413:84 , srv1509699090:84 , srv1518006394:84 , srv1524220810:84 , srv1560404237:84 , srv1604151659:84 , srv1632242373:84 , srv1691841015:84 , srv1703831557:84 , srv1707473424:84 , srv1712089142:84 , srv1728222862:84 , srv1747782612:84 , srv1761542066:84 , srv1773144917:84 , srv1847852533:84 , srv1852248465:84 , srv1871580138:84 , srv1929676688:84 , srv1950112136:84 , srv1974620710:84 , srv1974756577:84 , srv1980389129:84 , srv2026578456:84 , srv2040277732:84 , srv2044029384:84 , srv2065123265:84 , srv2071432451:84 , srv2103470022:84 , srv2117837303:84 , srv2128189028:84 , srv2130205288:84 , srv2139733538:84 , srv252526444:84 , srv270381081:84 , srv272884270:84 , srv284108103:84 , srv285628987:84 , srv295673423:84 , srv361254433:84 , srv366339969:84 , srv393210828:84 , srv436476017:84 , srv524901015:84 , srv529286076:84 , srv547410641:84 , srv579517430:84 , srv582285583:84 , srv582890445:84 , srv622221498:84 , srv644091570:84 , srv658218779:84 , srv665702047:84 , srv679678129:84 , srv685610725:84 , srv699872062:84 , srv72104483:84 , srv734389475:84 , srv74821594:84 , srv758512412:84 , srv771987511:84 , srv799775277:84 , srv820189990:84 , srv820445645:84 , srv820581874:84 , srv82631449:84 , srv84550751:84 , srv847749930:84 , srv872637492:84 , srv878034450:84 , srv894010756:84 , srv978095466:84 , srv1078946624:85 , srv1384702582:85 , srv1832566714:85 , srv1902863925:85 , srv34208514:85 , srv1695917394:684 }
2023-11-23 01:58:33,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-23 01:58:33,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv72104483=83, srv84550751=94, srv547410641=72, srv1524220810=24, srv1204114145=8, srv436476017=69, srv820445645=91, srv1712089142=33, srv1437460413=21, srv34208514=65, srv582285583=74, srv679678129=80, srv285628987=63, srv2040277732=49, srv1707473424=32, srv1428499488=20, srv1078946624=3, srv1334394762=12, srv622221498=76, srv1509699090=22, srv1847852533=39, srv582890445=75, srv295673423=64, srv1282214521=11, srv1037780728=1, srv1747782612=35, srv1695917394=30, srv699872062=82, srv361254433=66, srv1380367071=15, srv366339969=67, srv1728222862=34, srv847749930=95, srv1161201484=6, srv2130205288=56, srv284108103=62, srv1974620710=45, srv2026578456=48, srv1929676688=43, srv2044029384=50, srv1632242373=27, srv1518006394=23, srv1950112136=44, srv1238253673=9, srv1691841015=29, srv1408771750=17, srv1122593515=5, srv658218779=78, srv758512412=86, srv252526444=58, srv1761542066=36, srv1852248465=40, srv872637492=96, srv1902863925=42, srv894010756=98, srv1604151659=26, srv1871580138=41, srv2128189028=55, srv1267584770=10, srv1363626108=14, srv272884270=60, srv820581874=92, srv1384702582=16, srv1426233255=19, srv1064914756=2, srv1974756577=46, srv1120534152=4, srv133491636=13, srv2071432451=52, srv771987511=87, srv524901015=70, srv665702047=79, srv579517430=73, srv1421132009=18, srv2117837303=54, srv1773144917=37, srv1683467308=28, srv644091570=77, srv2103470022=53, srv273136805=61, srv1171723540=7, srv1980389129=47, srv74821594=85, srv2139733538=57, srv1560404237=25, srv1703831557=31, srv799775277=89, srv1016042292=0, srv734389475=84, srv82631449=93, srv820189990=90, srv2065123265=51, srv878034450=97, srv685610725=81, srv270381081=59, srv529286076=71, srv772570290=88, srv978095466=99, srv1832566714=38, srv393210828=68} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-23 01:58:33,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-23 01:58:33,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-23 01:58:33,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-23 01:58:33,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-23 01:58:33,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-23 01:58:33,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-23 01:58:33,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-23 01:58:33,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-23 01:58:33,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-23 01:58:33,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-23 01:58:33,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-23 01:58:33,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-23 01:58:33,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-23 01:58:33,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-23 01:58:33,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-23 01:58:33,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-23 01:58:33,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-23 01:58:33,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-23 01:58:33,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-23 01:58:33,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-23 01:58:33,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-23 01:58:33,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-23 01:58:33,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-23 01:58:33,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-23 01:58:33,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-23 01:58:33,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-23 01:58:33,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-23 01:58:33,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-23 01:58:33,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-23 01:58:33,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-23 01:58:33,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-23 01:58:33,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-23 01:58:33,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-23 01:58:33,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-23 01:58:33,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-23 01:58:33,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-23 01:58:33,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-23 01:58:33,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-23 01:58:33,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-23 01:58:33,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-23 01:58:33,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-23 01:58:33,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-23 01:58:33,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-23 01:58:33,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-23 01:58:33,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-23 01:58:33,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-23 01:58:33,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-23 01:58:33,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-23 01:58:33,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-23 01:58:33,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-23 01:58:33,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-23 01:58:33,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-23 01:58:33,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-23 01:58:33,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-23 01:58:33,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-23 01:58:33,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-23 01:58:33,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-23 01:58:33,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-23 01:58:33,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-23 01:58:33,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-23 01:58:33,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-23 01:58:33,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-23 01:58:33,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-23 01:58:33,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-23 01:58:33,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-23 01:58:33,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-23 01:58:33,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-23 01:58:33,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-23 01:58:33,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-23 01:58:33,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-23 01:58:33,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-23 01:58:33,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-23 01:58:33,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-23 01:58:33,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-23 01:58:33,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-23 01:58:33,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-23 01:58:33,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-23 01:58:33,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-23 01:58:33,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-23 01:58:34,089 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=8.793190239318795E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-23 01:58:34,089 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1120534152:83 , srv1238253673:83 , srv1683467308:83 , srv273136805:83 , srv772570290:83 , srv1016042292:84 , srv1037780728:84 , srv1064914756:84 , srv1122593515:84 , srv1161201484:84 , srv1171723540:84 , srv1204114145:84 , srv1267584770:84 , srv1282214521:84 , srv1334394762:84 , srv133491636:84 , srv1363626108:84 , srv1380367071:84 , srv1408771750:84 , srv1421132009:84 , srv1426233255:84 , srv1428499488:84 , srv1437460413:84 , srv1509699090:84 , srv1518006394:84 , srv1524220810:84 , srv1560404237:84 , srv1604151659:84 , srv1632242373:84 , srv1691841015:84 , srv1703831557:84 , srv1707473424:84 , srv1712089142:84 , srv1728222862:84 , srv1747782612:84 , srv1761542066:84 , srv1773144917:84 , srv1847852533:84 , srv1852248465:84 , srv1871580138:84 , srv1929676688:84 , srv1950112136:84 , srv1974620710:84 , srv1974756577:84 , srv1980389129:84 , srv2026578456:84 , srv2040277732:84 , srv2044029384:84 , srv2065123265:84 , srv2071432451:84 , srv2103470022:84 , srv2117837303:84 , srv2128189028:84 , srv2130205288:84 , srv2139733538:84 , srv252526444:84 , srv270381081:84 , srv272884270:84 , srv284108103:84 , srv285628987:84 , srv295673423:84 , srv361254433:84 , srv366339969:84 , srv393210828:84 , srv436476017:84 , srv524901015:84 , srv529286076:84 , srv547410641:84 , srv579517430:84 , srv582285583:84 , srv582890445:84 , srv622221498:84 , srv644091570:84 , srv658218779:84 , srv665702047:84 , srv679678129:84 , srv685610725:84 , srv699872062:84 , srv72104483:84 , srv734389475:84 , srv74821594:84 , srv758512412:84 , srv771987511:84 , srv799775277:84 , srv820189990:84 , srv820445645:84 , srv820581874:84 , srv82631449:84 , srv84550751:84 , srv847749930:84 , srv872637492:84 , srv878034450:84 , srv894010756:84 , srv978095466:84 , srv1078946624:85 , srv1384702582:85 , srv1832566714:85 , srv1902863925:85 , srv34208514:85 , srv1695917394:684 }
2023-11-23 01:58:34,115 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=490 (was 488) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 169), AvailableMemoryMB=12860 (was 13308)
2023-11-23 01:58:34,121 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=490, ProcessCount=167, AvailableMemoryMB=12859
2023-11-23 01:58:34,122 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-23 01:58:34,122 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-11-23 01:58:34,133 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1115169815:84 , srv1207241988:84 , srv1454657298:84 , srv1537378212:84 , srv1570989037:84 , srv1779823475:84 , srv1827939428:84 , srv1895300880:84 , srv1930603419:84 , srv1934489463:84 , srv1993445642:84 , srv2005633519:84 , srv2052423104:84 , srv241870906:84 , srv275753952:84 , srv364241358:84 , srv413382012:84 , srv478236086:84 , srv56007173:84 , srv568411827:84 , srv659175176:84 , srv725232899:84 , srv734103818:84 , srv748286905:84 , srv81440966:84 , srv815709056:84 , srv823692033:84 , srv834120823:84 , srv890899332:84 , srv152129288:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-11-23 01:58:34,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-23 01:58:34,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv734103818=23, srv815709056=26, srv568411827=20, srv1993445642=11, srv81440966=25, srv1779823475=6, srv1207241988=1, srv2052423104=13, srv364241358=16, srv1454657298=2, srv413382012=17, srv1570989037=5, srv152129288=3, srv1827939428=7, srv725232899=22, srv1115169815=0, srv275753952=15, srv56007173=19, srv834120823=28, srv890899332=29, srv1934489463=10, srv1537378212=4, srv1930603419=9, srv659175176=21, srv478236086=18, srv1895300880=8, srv823692033=27, srv2005633519=12, srv241870906=14, srv748286905=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-23 01:58:34,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-23 01:58:34,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-23 01:58:34,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-23 01:58:34,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-23 01:58:34,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-23 01:58:34,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-23 01:58:34,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-23 01:58:34,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-23 01:58:34,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-23 01:58:34,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-23 01:58:34,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-23 01:58:34,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-23 01:58:34,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-23 01:58:34,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-23 01:58:34,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-23 01:58:34,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-23 01:58:34,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-23 01:58:34,143 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-23 01:58:34,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-23 01:58:34,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-23 01:58:34,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-23 01:58:34,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-23 01:58:34,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-23 01:58:34,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-23 01:58:34,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-23 01:58:34,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-23 01:58:34,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-23 01:58:34,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-23 01:58:34,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-23 01:58:34,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-23 01:58:34,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-23 01:58:34,157 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.085296734975988, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-23 01:58:34,157 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912461810490995, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.085296734975988, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-11-23 02:00:34,147 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9983957 different iterations.  Found a solution that moves 1839 regions; Going from a computed imbalance of 0.9912461810490995 to a new imbalance of 4.292317741546655E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6811111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-23 02:00:34,152 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1115169815:90 , srv1207241988:90 , srv1454657298:90 , srv152129288:90 , srv1537378212:90 , srv1570989037:90 , srv1779823475:90 , srv1827939428:90 , srv1895300880:90 , srv1930603419:90 , srv1934489463:90 , srv1993445642:90 , srv2005633519:90 , srv2052423104:90 , srv241870906:90 , srv275753952:90 , srv364241358:90 , srv413382012:90 , srv478236086:90 , srv56007173:90 , srv568411827:90 , srv659175176:90 , srv725232899:90 , srv734103818:90 , srv748286905:90 , srv81440966:90 , srv815709056:90 , srv823692033:90 , srv834120823:90 , srv890899332:90 }
2023-11-23 02:00:34,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-23 02:00:34,153 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv734103818=23, srv815709056=26, srv568411827=20, srv1993445642=11, srv81440966=25, srv1779823475=6, srv1207241988=1, srv2052423104=13, srv364241358=16, srv1454657298=2, srv413382012=17, srv1570989037=5, srv152129288=3, srv1827939428=7, srv725232899=22, srv1115169815=0, srv275753952=15, srv56007173=19, srv834120823=28, srv890899332=29, srv1934489463=10, srv1537378212=4, srv1930603419=9, srv659175176=21, srv478236086=18, srv1895300880=8, srv823692033=27, srv2005633519=12, srv241870906=14, srv748286905=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-23 02:00:34,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-23 02:00:34,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-23 02:00:34,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-23 02:00:34,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-23 02:00:34,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-23 02:00:34,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-23 02:00:34,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-23 02:00:34,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-23 02:00:34,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-23 02:00:34,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-23 02:00:34,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-23 02:00:34,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-23 02:00:34,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-23 02:00:34,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-23 02:00:34,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-23 02:00:34,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-23 02:00:34,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-23 02:00:34,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-23 02:00:34,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-23 02:00:34,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-23 02:00:34,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-23 02:00:34,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-23 02:00:34,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-23 02:00:34,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-23 02:00:34,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-23 02:00:34,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-23 02:00:34,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-23 02:00:34,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-23 02:00:34,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-23 02:00:34,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-23 02:00:34,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-23 02:00:34,168 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-23 02:00:34,168 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1115169815:90 , srv1207241988:90 , srv1454657298:90 , srv152129288:90 , srv1537378212:90 , srv1570989037:90 , srv1779823475:90 , srv1827939428:90 , srv1895300880:90 , srv1930603419:90 , srv1934489463:90 , srv1993445642:90 , srv2005633519:90 , srv2052423104:90 , srv241870906:90 , srv275753952:90 , srv364241358:90 , srv413382012:90 , srv478236086:90 , srv56007173:90 , srv568411827:90 , srv659175176:90 , srv725232899:90 , srv734103818:90 , srv748286905:90 , srv81440966:90 , srv815709056:90 , srv823692033:90 , srv834120823:90 , srv890899332:90 }
2023-11-23 02:00:34,179 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=163 (was 490), ProcessCount=165 (was 167), AvailableMemoryMB=14032 (was 12859) - AvailableMemoryMB LEAK? -