2023-11-02 02:55:33,305 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-11-02 02:55:33,588 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-11-02 02:55:33,687 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-11-02 02:55:33,744 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-02 02:55:33,751 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-02 02:55:33,763 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=114, ProcessCount=169, AvailableMemoryMB=11905
2023-11-02 02:55:33,765 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-02 02:55:33,765 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-02 02:55:33,875 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1058623324:84 , srv1062039145:84 , srv1133487801:84 , srv1137871259:84 , srv1142111283:84 , srv1146449825:84 , srv122304465:84 , srv1275365121:84 , srv129442513:84 , srv1317074381:84 , srv1377224996:84 , srv1428956008:84 , srv1448932890:84 , srv1461287072:84 , srv1470311891:84 , srv1485575702:84 , srv150066947:84 , srv1516456189:84 , srv1530097450:84 , srv1537897131:84 , srv1556251875:84 , srv1558056590:84 , srv1634408544:84 , srv1663139907:84 , srv1672027413:84 , srv1679102491:84 , srv1681976318:84 , srv1714549420:84 , srv1718229461:84 , srv1720261349:84 , srv1732224713:84 , srv1747144321:84 , srv175288175:84 , srv1772602161:84 , srv178245889:84 , srv1798804720:84 , srv1799705468:84 , srv1834320772:84 , srv1848065831:84 , srv1851129693:84 , srv1862358562:84 , srv1876634200:84 , srv1899729473:84 , srv1900407428:84 , srv1915324676:84 , srv1916508018:84 , srv1917035507:84 , srv1968066532:84 , srv1978370587:84 , srv1990388669:84 , srv2000122556:84 , srv2005296181:84 , srv2012670408:84 , srv2023127801:84 , srv2036614892:84 , srv2039760349:84 , srv2055677584:84 , srv206889882:84 , srv2074077803:84 , srv2075435657:84 , srv2091204350:84 , srv2104607007:84 , srv2116463846:84 , srv2116759911:84 , srv2129336500:84 , srv2137657374:84 , srv2145935532:84 , srv264585224:84 , srv307236757:84 , srv32966871:84 , srv357102613:84 , srv365177051:84 , srv372013614:84 , srv392707397:84 , srv396716796:84 , srv448176549:84 , srv459990838:84 , srv523179060:84 , srv542706559:84 , srv557505520:84 , srv579161596:84 , srv587124937:84 , srv626878719:84 , srv636886285:84 , srv637110189:84 , srv750239678:84 , srv791758446:84 , srv811639251:84 , srv818162561:84 , srv825917864:84 , srv851076987:84 , srv857744474:84 , srv860372460:84 , srv869100116:84 , srv927085486:84 , srv941332986:84 , srv96927222:84 , srv976721895:84 , srv990955342:84 , srv733222103:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-11-02 02:55:33,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-02 02:55:33,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2074077803=58, srv1851129693=39, srv1915324676=44, srv1862358562=40, srv2039760349=55, srv392707397=73, srv1917035507=46, srv1978370587=48, srv1058623324=0, srv1720261349=29, srv1900407428=43, srv1137871259=3, srv2145935532=66, srv122304465=6, srv1537897131=19, srv811639251=88, srv1672027413=24, srv1634408544=22, srv1448932890=12, srv2116463846=62, srv1718229461=28, srv1848065831=38, srv733222103=85, srv1799705468=36, srv1556251875=20, srv150066947=16, srv2000122556=50, srv1876634200=41, srv626878719=82, srv1663139907=23, srv2104607007=61, srv1990388669=49, srv818162561=89, srv1530097450=18, srv175288175=32, srv2005296181=51, srv448176549=75, srv851076987=91, srv1916508018=45, srv860372460=93, srv1968066532=47, srv1732224713=30, srv2137657374=65, srv96927222=97, srv1275365121=7, srv1428956008=11, srv1485575702=15, srv396716796=74, srv459990838=76, srv1714549420=27, srv1899729473=42, srv1317074381=9, srv523179060=77, srv2036614892=54, srv579161596=80, srv206889882=57, srv1062039145=1, srv857744474=92, srv2116759911=63, srv264585224=67, srv1747144321=31, srv2055677584=56, srv1772602161=33, srv941332986=96, srv825917864=90, srv1516456189=17, srv869100116=94, srv129442513=8, srv1681976318=26, srv2012670408=52, srv976721895=98, srv637110189=84, srv1798804720=35, srv357102613=70, srv307236757=68, srv750239678=86, srv1377224996=10, srv927085486=95, srv372013614=72, srv791758446=87, srv557505520=79, srv1558056590=21, srv2129336500=64, srv990955342=99, srv1470311891=14, srv636886285=83, srv1461287072=13, srv2023127801=53, srv1834320772=37, srv2075435657=59, srv1679102491=25, srv32966871=69, srv542706559=78, srv1146449825=5, srv1133487801=2, srv1142111283=4, srv2091204350=60, srv178245889=34, srv587124937=81, srv365177051=71} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-02 02:55:33,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-02 02:55:33,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-02 02:55:33,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-02 02:55:33,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-02 02:55:33,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-02 02:55:33,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-02 02:55:33,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-02 02:55:33,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-02 02:55:33,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-02 02:55:33,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-02 02:55:33,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-02 02:55:33,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-02 02:55:33,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-02 02:55:33,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-02 02:55:33,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-02 02:55:33,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-02 02:55:33,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-02 02:55:33,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-02 02:55:33,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-02 02:55:33,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-02 02:55:33,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-02 02:55:33,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-02 02:55:33,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-02 02:55:33,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-02 02:55:33,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-02 02:55:33,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-02 02:55:33,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-02 02:55:33,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-02 02:55:33,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-02 02:55:33,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-02 02:55:33,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-02 02:55:33,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-02 02:55:33,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-02 02:55:33,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-02 02:55:33,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-02 02:55:33,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-02 02:55:33,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-02 02:55:33,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-02 02:55:33,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-02 02:55:33,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-02 02:55:33,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-02 02:55:33,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-02 02:55:33,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-02 02:55:33,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-02 02:55:33,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-02 02:55:33,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-02 02:55:33,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-02 02:55:33,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-02 02:55:33,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-02 02:55:33,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-02 02:55:33,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-02 02:55:33,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-02 02:55:33,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-02 02:55:33,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-02 02:55:33,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-02 02:55:33,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-02 02:55:33,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-02 02:55:33,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-02 02:55:33,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-02 02:55:33,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-02 02:55:33,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-02 02:55:33,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-02 02:55:33,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-02 02:55:33,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-02 02:55:33,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-02 02:55:33,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-02 02:55:33,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-02 02:55:33,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-02 02:55:33,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-02 02:55:33,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-02 02:55:33,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-02 02:55:33,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-02 02:55:33,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-02 02:55:33,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-02 02:55:33,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-02 02:55:33,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-02 02:55:33,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-02 02:55:33,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-02 02:55:33,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-02 02:55:33,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-02 02:55:33,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-02 02:55:33,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-02 02:55:33,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-02 02:55:33,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-02 02:55:33,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-02 02:55:33,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-02 02:55:33,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-02 02:55:33,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-02 02:55:33,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-02 02:55:33,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-02 02:55:33,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-02 02:55:33,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-02 02:55:33,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-02 02:55:33,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-02 02:55:33,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-02 02:55:33,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-02 02:55:33,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-02 02:55:33,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-02 02:55:33,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-02 02:55:33,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-02 02:55:33,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-02 02:55:34,177 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 02:55:34,177 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-02 02:55:34,177 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-02 02:55:34,624 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 618 ms to try 5000 different iterations.  Found a solution that moves 4743 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17524148745668838. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03957342059902668, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05587584441220245, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.527, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03957342059902668, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17016666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23958333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 02:55:34,640 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1862358562:82 , srv2023127801:83 , srv2075435657:83 , srv1058623324:84 , srv1062039145:84 , srv1556251875:84 , srv1634408544:84 , srv1681976318:84 , srv1772602161:84 , srv178245889:84 , srv1834320772:84 , srv1917035507:84 , srv1990388669:84 , srv2039760349:84 , srv307236757:84 , srv32966871:84 , srv579161596:84 , srv96927222:84 , srv1377224996:85 , srv1530097450:85 , srv1537897131:85 , srv1672027413:85 , srv175288175:85 , srv1848065831:85 , srv1899729473:85 , srv1968066532:85 , srv206889882:85 , srv2116759911:85 , srv2129336500:85 , srv2137657374:85 , srv372013614:85 , srv392707397:85 , srv396716796:85 , srv750239678:85 , srv818162561:85 , srv869100116:85 , srv1142111283:86 , srv122304465:86 , srv1275365121:86 , srv1428956008:86 , srv1470311891:86 , srv1485575702:86 , srv150066947:86 , srv1679102491:86 , srv1720261349:86 , srv1747144321:86 , srv1798804720:86 , srv1900407428:86 , srv1978370587:86 , srv2036614892:86 , srv2074077803:86 , srv626878719:86 , srv637110189:86 , srv791758446:86 , srv825917864:86 , srv860372460:86 , srv1133487801:87 , srv1137871259:87 , srv1461287072:87 , srv1516456189:87 , srv1799705468:87 , srv1915324676:87 , srv2005296181:87 , srv2104607007:87 , srv2145935532:87 , srv264585224:87 , srv357102613:87 , srv636886285:87 , srv811639251:87 , srv941332986:87 , srv976721895:87 , srv1146449825:88 , srv129442513:88 , srv1851129693:88 , srv1916508018:88 , srv2000122556:88 , srv2012670408:88 , srv448176549:88 , srv557505520:88 , srv851076987:88 , srv857744474:88 , srv927085486:88 , srv990955342:88 , srv1317074381:89 , srv1448932890:89 , srv1558056590:89 , srv1663139907:89 , srv2116463846:89 , srv365177051:89 , srv523179060:89 , srv542706559:89 , srv1714549420:90 , srv2055677584:90 , srv2091204350:90 , srv459990838:90 , srv1732224713:91 , srv1876634200:91 , srv587124937:92 , srv1718229461:93 , srv733222103:442 }
2023-11-02 02:55:34,647 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-02 02:55:34,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2074077803=58, srv1851129693=39, srv1915324676=44, srv1862358562=40, srv2039760349=55, srv392707397=73, srv1917035507=46, srv1978370587=48, srv1058623324=0, srv1720261349=29, srv1900407428=43, srv1137871259=3, srv2145935532=66, srv122304465=6, srv1537897131=19, srv811639251=88, srv1672027413=24, srv1634408544=22, srv1448932890=12, srv2116463846=62, srv1718229461=28, srv1848065831=38, srv733222103=85, srv1799705468=36, srv1556251875=20, srv150066947=16, srv2000122556=50, srv1876634200=41, srv626878719=82, srv1663139907=23, srv2104607007=61, srv1990388669=49, srv818162561=89, srv1530097450=18, srv175288175=32, srv2005296181=51, srv448176549=75, srv851076987=91, srv1916508018=45, srv860372460=93, srv1968066532=47, srv1732224713=30, srv2137657374=65, srv96927222=97, srv1275365121=7, srv1428956008=11, srv1485575702=15, srv396716796=74, srv459990838=76, srv1714549420=27, srv1899729473=42, srv1317074381=9, srv523179060=77, srv2036614892=54, srv579161596=80, srv206889882=57, srv1062039145=1, srv857744474=92, srv2116759911=63, srv264585224=67, srv1747144321=31, srv2055677584=56, srv1772602161=33, srv941332986=96, srv825917864=90, srv1516456189=17, srv869100116=94, srv129442513=8, srv1681976318=26, srv2012670408=52, srv976721895=98, srv637110189=84, srv1798804720=35, srv357102613=70, srv307236757=68, srv750239678=86, srv1377224996=10, srv927085486=95, srv372013614=72, srv791758446=87, srv557505520=79, srv1558056590=21, srv2129336500=64, srv990955342=99, srv1470311891=14, srv636886285=83, srv1461287072=13, srv2023127801=53, srv1834320772=37, srv2075435657=59, srv1679102491=25, srv32966871=69, srv542706559=78, srv1146449825=5, srv1133487801=2, srv1142111283=4, srv2091204350=60, srv178245889=34, srv587124937=81, srv365177051=71} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-02 02:55:34,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-02 02:55:34,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-02 02:55:34,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-02 02:55:34,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-02 02:55:34,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-02 02:55:34,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-02 02:55:34,673 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-02 02:55:34,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-02 02:55:34,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-02 02:55:34,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-02 02:55:34,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-02 02:55:34,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-02 02:55:34,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-02 02:55:34,674 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-02 02:55:34,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-02 02:55:34,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-02 02:55:34,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-02 02:55:34,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-02 02:55:34,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-02 02:55:34,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-02 02:55:34,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-02 02:55:34,675 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-02 02:55:34,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-02 02:55:34,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-02 02:55:34,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-02 02:55:34,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-02 02:55:34,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-02 02:55:34,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-02 02:55:34,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-02 02:55:34,676 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-02 02:55:34,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-02 02:55:34,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-02 02:55:34,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-02 02:55:34,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-02 02:55:34,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-02 02:55:34,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-02 02:55:34,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-02 02:55:34,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-02 02:55:34,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-02 02:55:34,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-02 02:55:34,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-02 02:55:34,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-02 02:55:34,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-02 02:55:34,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-02 02:55:34,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-02 02:55:34,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-02 02:55:34,678 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-02 02:55:34,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-02 02:55:34,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-02 02:55:34,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-02 02:55:34,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-02 02:55:34,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-02 02:55:34,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-02 02:55:34,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-02 02:55:34,679 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-02 02:55:34,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-02 02:55:34,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-02 02:55:34,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-02 02:55:34,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-02 02:55:34,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-02 02:55:34,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-02 02:55:34,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-02 02:55:34,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-02 02:55:34,680 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-02 02:55:34,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-02 02:55:34,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-02 02:55:34,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-02 02:55:34,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-02 02:55:34,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-02 02:55:34,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-02 02:55:34,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-02 02:55:34,681 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-02 02:55:34,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-02 02:55:34,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-02 02:55:34,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-02 02:55:34,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-02 02:55:34,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-02 02:55:34,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-02 02:55:34,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-02 02:55:34,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-02 02:55:34,682 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-02 02:55:34,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-02 02:55:34,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-02 02:55:34,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-02 02:55:34,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-02 02:55:34,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-02 02:55:34,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-02 02:55:34,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-02 02:55:34,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-02 02:55:34,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-02 02:55:34,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-02 02:55:34,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-02 02:55:34,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-02 02:55:34,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-02 02:55:34,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-02 02:55:34,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-02 02:55:34,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-02 02:55:34,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-02 02:55:34,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-02 02:55:34,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-02 02:55:34,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-02 02:55:34,870 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03957342059902668, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05587584441220245, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03957342059902668, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17016666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23958333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 02:55:34,870 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-02 02:55:34,871 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17520827626085128, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03957342059902668, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05587584441220245, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03957342059902668, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17016666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23958333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-02 02:55:35,130 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 417 ms to try 5000 different iterations.  Found a solution that moves 2596 regions; Going from a computed imbalance of 0.17520827626085128 to a new imbalance of 0.014894483200037261. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020371900768957882); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031366198109404074, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28844444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020371900768957882); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042583333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 02:55:35,142 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1718229461:80 , srv587124937:80 , srv459990838:81 , srv1714549420:82 , srv1732224713:82 , srv1876634200:82 , srv1916508018:82 , srv2055677584:82 , srv365177051:82 , srv1133487801:83 , srv129442513:83 , srv1663139907:83 , srv2000122556:83 , srv2005296181:83 , srv448176549:83 , srv523179060:83 , srv557505520:83 , srv811639251:83 , srv1317074381:84 , srv1448932890:84 , srv150066947:84 , srv1516456189:84 , srv1558056590:84 , srv1851129693:84 , srv2012670408:84 , srv2091204350:84 , srv2104607007:84 , srv2145935532:84 , srv542706559:84 , srv851076987:84 , srv857744474:84 , srv860372460:84 , srv941332986:84 , srv990955342:84 , srv1137871259:85 , srv1720261349:85 , srv1915324676:85 , srv2116463846:85 , srv2116759911:85 , srv264585224:85 , srv626878719:85 , srv818162561:85 , srv1146449825:86 , srv1428956008:86 , srv1485575702:86 , srv1537897131:86 , srv1672027413:86 , srv1834320772:86 , srv1899729473:86 , srv1978370587:86 , srv1990388669:86 , srv206889882:86 , srv2074077803:86 , srv307236757:86 , srv372013614:86 , srv392707397:86 , srv636886285:86 , srv791758446:86 , srv927085486:86 , srv122304465:87 , srv1275365121:87 , srv1377224996:87 , srv1470311891:87 , srv1634408544:87 , srv1679102491:87 , srv1681976318:87 , srv1747144321:87 , srv178245889:87 , srv1799705468:87 , srv1900407428:87 , srv2036614892:87 , srv2137657374:87 , srv357102613:87 , srv396716796:87 , srv579161596:87 , srv976721895:87 , srv1058623324:88 , srv1062039145:88 , srv1142111283:88 , srv1461287072:88 , srv175288175:88 , srv1772602161:88 , srv1798804720:88 , srv1848065831:88 , srv1917035507:88 , srv1968066532:88 , srv2023127801:88 , srv2129336500:88 , srv637110189:88 , srv825917864:88 , srv869100116:88 , srv1530097450:89 , srv32966871:89 , srv750239678:89 , srv1862358562:90 , srv2075435657:90 , srv1556251875:91 , srv2039760349:92 , srv96927222:92 , srv733222103:513 }
2023-11-02 02:55:35,146 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-02 02:55:35,147 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2074077803=58, srv1851129693=39, srv1915324676=44, srv1862358562=40, srv2039760349=55, srv392707397=73, srv1917035507=46, srv1978370587=48, srv1058623324=0, srv1720261349=29, srv1900407428=43, srv1137871259=3, srv2145935532=66, srv122304465=6, srv1537897131=19, srv811639251=88, srv1672027413=24, srv1634408544=22, srv1448932890=12, srv2116463846=62, srv1718229461=28, srv1848065831=38, srv733222103=85, srv1799705468=36, srv1556251875=20, srv150066947=16, srv2000122556=50, srv1876634200=41, srv626878719=82, srv1663139907=23, srv2104607007=61, srv1990388669=49, srv818162561=89, srv1530097450=18, srv175288175=32, srv2005296181=51, srv448176549=75, srv851076987=91, srv1916508018=45, srv860372460=93, srv1968066532=47, srv1732224713=30, srv2137657374=65, srv96927222=97, srv1275365121=7, srv1428956008=11, srv1485575702=15, srv396716796=74, srv459990838=76, srv1714549420=27, srv1899729473=42, srv1317074381=9, srv523179060=77, srv2036614892=54, srv579161596=80, srv206889882=57, srv1062039145=1, srv857744474=92, srv2116759911=63, srv264585224=67, srv1747144321=31, srv2055677584=56, srv1772602161=33, srv941332986=96, srv825917864=90, srv1516456189=17, srv869100116=94, srv129442513=8, srv1681976318=26, srv2012670408=52, srv976721895=98, srv637110189=84, srv1798804720=35, srv357102613=70, srv307236757=68, srv750239678=86, srv1377224996=10, srv927085486=95, srv372013614=72, srv791758446=87, srv557505520=79, srv1558056590=21, srv2129336500=64, srv990955342=99, srv1470311891=14, srv636886285=83, srv1461287072=13, srv2023127801=53, srv1834320772=37, srv2075435657=59, srv1679102491=25, srv32966871=69, srv542706559=78, srv1146449825=5, srv1133487801=2, srv1142111283=4, srv2091204350=60, srv178245889=34, srv587124937=81, srv365177051=71} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-02 02:55:35,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-02 02:55:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-02 02:55:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-02 02:55:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-02 02:55:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-02 02:55:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-02 02:55:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-02 02:55:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-02 02:55:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-02 02:55:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-02 02:55:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-02 02:55:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-02 02:55:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-02 02:55:35,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-02 02:55:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-02 02:55:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-02 02:55:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-02 02:55:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-02 02:55:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-02 02:55:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-02 02:55:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-02 02:55:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-02 02:55:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-02 02:55:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-02 02:55:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-02 02:55:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-02 02:55:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-02 02:55:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-02 02:55:35,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-02 02:55:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-02 02:55:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-02 02:55:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-02 02:55:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-02 02:55:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-02 02:55:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-02 02:55:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-02 02:55:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-02 02:55:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-02 02:55:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-02 02:55:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-02 02:55:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-02 02:55:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-02 02:55:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-02 02:55:35,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-02 02:55:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-02 02:55:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-02 02:55:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-02 02:55:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-02 02:55:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-02 02:55:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-02 02:55:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-02 02:55:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-02 02:55:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-02 02:55:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-02 02:55:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-02 02:55:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-02 02:55:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-02 02:55:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-02 02:55:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-02 02:55:35,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-02 02:55:35,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-02 02:55:35,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-02 02:55:35,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-02 02:55:35,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-02 02:55:35,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-02 02:55:35,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-02 02:55:35,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-02 02:55:35,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-02 02:55:35,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-02 02:55:35,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-02 02:55:35,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-02 02:55:35,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-02 02:55:35,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-02 02:55:35,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-02 02:55:35,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-02 02:55:35,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-02 02:55:35,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-02 02:55:35,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-02 02:55:35,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-02 02:55:35,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-02 02:55:35,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-02 02:55:35,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-02 02:55:35,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-02 02:55:35,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-02 02:55:35,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-02 02:55:35,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-02 02:55:35,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-02 02:55:35,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-02 02:55:35,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-02 02:55:35,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-02 02:55:35,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-02 02:55:35,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-02 02:55:35,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-02 02:55:35,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-02 02:55:35,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-02 02:55:35,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-02 02:55:35,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-02 02:55:35,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-02 02:55:35,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-02 02:55:35,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-02 02:55:35,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-02 02:55:35,341 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020371900768957882); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031366198109404074, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020371900768957882); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042583333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 02:55:35,341 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-02 02:55:35,341 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014876305619520042, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020371900768957882); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.031366198109404074, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020371900768957882); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042583333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-02 02:55:35,590 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 391 ms to try 5000 different iterations.  Found a solution that moves 929 regions; Going from a computed imbalance of 0.014876305619520042 to a new imbalance of 5.62960451627693E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006217570812329759); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008632670957261212); MoveCostFunction : (multiplier=7.0, imbalance=0.10322222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006217570812329759); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 02:55:35,591 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2039760349:81 , srv96927222:81 , srv1461287072:82 , srv1142111283:83 , srv1146449825:83 , srv1556251875:83 , srv1747144321:83 , srv1798804720:83 , srv1799705468:83 , srv1968066532:83 , srv2116463846:83 , srv357102613:83 , srv927085486:83 , srv1058623324:84 , srv1275365121:84 , srv1317074381:84 , srv1428956008:84 , srv1448932890:84 , srv1530097450:84 , srv1558056590:84 , srv1679102491:84 , srv1876634200:84 , srv2036614892:84 , srv2091204350:84 , srv32966871:84 , srv396716796:84 , srv542706559:84 , srv637110189:84 , srv750239678:84 , srv869100116:84 , srv976721895:84 , srv1062039145:85 , srv1137871259:85 , srv122304465:85 , srv1377224996:85 , srv1470311891:85 , srv1714549420:85 , srv1718229461:85 , srv1732224713:85 , srv175288175:85 , srv1848065831:85 , srv1851129693:85 , srv1862358562:85 , srv1900407428:85 , srv1917035507:85 , srv206889882:85 , srv2075435657:85 , srv2129336500:85 , srv2137657374:85 , srv264585224:85 , srv523179060:85 , srv587124937:85 , srv636886285:85 , srv825917864:85 , srv990955342:85 , srv1516456189:86 , srv1663139907:86 , srv1672027413:86 , srv1681976318:86 , srv1720261349:86 , srv1772602161:86 , srv1915324676:86 , srv1978370587:86 , srv2012670408:86 , srv2023127801:86 , srv2055677584:86 , srv392707397:86 , srv448176549:86 , srv459990838:86 , srv557505520:86 , srv791758446:86 , srv851076987:86 , srv857744474:86 , srv941332986:86 , srv1133487801:87 , srv129442513:87 , srv1485575702:87 , srv150066947:87 , srv1537897131:87 , srv1634408544:87 , srv178245889:87 , srv1899729473:87 , srv1990388669:87 , srv2000122556:87 , srv2005296181:87 , srv2074077803:87 , srv2104607007:87 , srv2116759911:87 , srv2145935532:87 , srv372013614:87 , srv579161596:87 , srv626878719:87 , srv1834320772:88 , srv1916508018:88 , srv307236757:88 , srv365177051:88 , srv811639251:88 , srv818162561:88 , srv860372460:88 , srv733222103:558 }
2023-11-02 02:55:35,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-02 02:55:35,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2074077803=58, srv1851129693=39, srv1915324676=44, srv1862358562=40, srv2039760349=55, srv392707397=73, srv1917035507=46, srv1978370587=48, srv1058623324=0, srv1720261349=29, srv1900407428=43, srv1137871259=3, srv2145935532=66, srv122304465=6, srv1537897131=19, srv811639251=88, srv1672027413=24, srv1634408544=22, srv1448932890=12, srv2116463846=62, srv1718229461=28, srv1848065831=38, srv733222103=85, srv1799705468=36, srv1556251875=20, srv150066947=16, srv2000122556=50, srv1876634200=41, srv626878719=82, srv1663139907=23, srv2104607007=61, srv1990388669=49, srv818162561=89, srv1530097450=18, srv175288175=32, srv2005296181=51, srv448176549=75, srv851076987=91, srv1916508018=45, srv860372460=93, srv1968066532=47, srv1732224713=30, srv2137657374=65, srv96927222=97, srv1275365121=7, srv1428956008=11, srv1485575702=15, srv396716796=74, srv459990838=76, srv1714549420=27, srv1899729473=42, srv1317074381=9, srv523179060=77, srv2036614892=54, srv579161596=80, srv206889882=57, srv1062039145=1, srv857744474=92, srv2116759911=63, srv264585224=67, srv1747144321=31, srv2055677584=56, srv1772602161=33, srv941332986=96, srv825917864=90, srv1516456189=17, srv869100116=94, srv129442513=8, srv1681976318=26, srv2012670408=52, srv976721895=98, srv637110189=84, srv1798804720=35, srv357102613=70, srv307236757=68, srv750239678=86, srv1377224996=10, srv927085486=95, srv372013614=72, srv791758446=87, srv557505520=79, srv1558056590=21, srv2129336500=64, srv990955342=99, srv1470311891=14, srv636886285=83, srv1461287072=13, srv2023127801=53, srv1834320772=37, srv2075435657=59, srv1679102491=25, srv32966871=69, srv542706559=78, srv1146449825=5, srv1133487801=2, srv1142111283=4, srv2091204350=60, srv178245889=34, srv587124937=81, srv365177051=71} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-02 02:55:35,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-02 02:55:35,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-02 02:55:35,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-02 02:55:35,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-02 02:55:35,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-02 02:55:35,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-02 02:55:35,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-02 02:55:35,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-02 02:55:35,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-02 02:55:35,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-02 02:55:35,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-02 02:55:35,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-02 02:55:35,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-02 02:55:35,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-02 02:55:35,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-02 02:55:35,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-02 02:55:35,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-02 02:55:35,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-02 02:55:35,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-02 02:55:35,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-02 02:55:35,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-02 02:55:35,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-02 02:55:35,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-02 02:55:35,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-02 02:55:35,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-02 02:55:35,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-02 02:55:35,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-02 02:55:35,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-02 02:55:35,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-02 02:55:35,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-02 02:55:35,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-02 02:55:35,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-02 02:55:35,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-02 02:55:35,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-02 02:55:35,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-02 02:55:35,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-02 02:55:35,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-02 02:55:35,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-02 02:55:35,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-02 02:55:35,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-02 02:55:35,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-02 02:55:35,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-02 02:55:35,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-02 02:55:35,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-02 02:55:35,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-02 02:55:35,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-02 02:55:35,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-02 02:55:35,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-02 02:55:35,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-02 02:55:35,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-02 02:55:35,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-02 02:55:35,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-02 02:55:35,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-02 02:55:35,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-02 02:55:35,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-02 02:55:35,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-02 02:55:35,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-02 02:55:35,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-02 02:55:35,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-02 02:55:35,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-02 02:55:35,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-02 02:55:35,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-02 02:55:35,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-02 02:55:35,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-02 02:55:35,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-02 02:55:35,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-02 02:55:35,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-02 02:55:35,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-02 02:55:35,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-02 02:55:35,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-02 02:55:35,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-02 02:55:35,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-02 02:55:35,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-02 02:55:35,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-02 02:55:35,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-02 02:55:35,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-02 02:55:35,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-02 02:55:35,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-02 02:55:35,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-02 02:55:35,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-02 02:55:35,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-02 02:55:35,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-02 02:55:35,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-02 02:55:35,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-02 02:55:35,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-02 02:55:35,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-02 02:55:35,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-02 02:55:35,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-02 02:55:35,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-02 02:55:35,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-02 02:55:35,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-02 02:55:35,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-02 02:55:35,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-02 02:55:35,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-02 02:55:35,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-02 02:55:35,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-02 02:55:35,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-02 02:55:35,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-02 02:55:35,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-02 02:55:35,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-02 02:55:35,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-02 02:55:35,735 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006217570812329759); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008632670957261212); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006217570812329759); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 02:55:35,735 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-02 02:55:35,735 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.564554545936035E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006217570812329759); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008632670957261212); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006217570812329759); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-02 02:55:36,094 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 467 ms to try 5000 different iterations.  Found a solution that moves 240 regions; Going from a computed imbalance of 5.564554545936035E-4 to a new imbalance of 1.9752236050470423E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.02666666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 02:55:36,095 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1485575702:83 , srv1772602161:83 , srv2074077803:83 , srv365177051:83 , srv825917864:83 , srv1062039145:84 , srv122304465:84 , srv129442513:84 , srv1470311891:84 , srv1530097450:84 , srv1537897131:84 , srv1556251875:84 , srv1634408544:84 , srv1663139907:84 , srv1718229461:84 , srv1732224713:84 , srv175288175:84 , srv178245889:84 , srv1834320772:84 , srv1848065831:84 , srv1899729473:84 , srv1900407428:84 , srv1916508018:84 , srv1978370587:84 , srv2000122556:84 , srv2012670408:84 , srv2023127801:84 , srv2055677584:84 , srv2075435657:84 , srv2091204350:84 , srv2104607007:84 , srv2129336500:84 , srv2137657374:84 , srv2145935532:84 , srv307236757:84 , srv372013614:84 , srv579161596:84 , srv626878719:84 , srv636886285:84 , srv637110189:84 , srv750239678:84 , srv791758446:84 , srv818162561:84 , srv851076987:84 , srv857744474:84 , srv860372460:84 , srv976721895:84 , srv1133487801:85 , srv1137871259:85 , srv1142111283:85 , srv1146449825:85 , srv1275365121:85 , srv1317074381:85 , srv1377224996:85 , srv1448932890:85 , srv1461287072:85 , srv150066947:85 , srv1516456189:85 , srv1558056590:85 , srv1672027413:85 , srv1679102491:85 , srv1681976318:85 , srv1714549420:85 , srv1720261349:85 , srv1798804720:85 , srv1799705468:85 , srv1851129693:85 , srv1862358562:85 , srv1876634200:85 , srv1915324676:85 , srv1917035507:85 , srv1990388669:85 , srv2005296181:85 , srv2036614892:85 , srv2039760349:85 , srv2116463846:85 , srv2116759911:85 , srv264585224:85 , srv32966871:85 , srv392707397:85 , srv448176549:85 , srv459990838:85 , srv523179060:85 , srv542706559:85 , srv557505520:85 , srv811639251:85 , srv869100116:85 , srv927085486:85 , srv941332986:85 , srv96927222:85 , srv990955342:85 , srv1058623324:86 , srv1428956008:86 , srv1747144321:86 , srv1968066532:86 , srv206889882:86 , srv357102613:86 , srv396716796:86 , srv587124937:86 , srv733222103:629 }
2023-11-02 02:55:36,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-02 02:55:36,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2074077803=58, srv1851129693=39, srv1915324676=44, srv1862358562=40, srv2039760349=55, srv392707397=73, srv1917035507=46, srv1978370587=48, srv1058623324=0, srv1720261349=29, srv1900407428=43, srv1137871259=3, srv2145935532=66, srv122304465=6, srv1537897131=19, srv811639251=88, srv1672027413=24, srv1634408544=22, srv1448932890=12, srv2116463846=62, srv1718229461=28, srv1848065831=38, srv733222103=85, srv1799705468=36, srv1556251875=20, srv150066947=16, srv2000122556=50, srv1876634200=41, srv626878719=82, srv1663139907=23, srv2104607007=61, srv1990388669=49, srv818162561=89, srv1530097450=18, srv175288175=32, srv2005296181=51, srv448176549=75, srv851076987=91, srv1916508018=45, srv860372460=93, srv1968066532=47, srv1732224713=30, srv2137657374=65, srv96927222=97, srv1275365121=7, srv1428956008=11, srv1485575702=15, srv396716796=74, srv459990838=76, srv1714549420=27, srv1899729473=42, srv1317074381=9, srv523179060=77, srv2036614892=54, srv579161596=80, srv206889882=57, srv1062039145=1, srv857744474=92, srv2116759911=63, srv264585224=67, srv1747144321=31, srv2055677584=56, srv1772602161=33, srv941332986=96, srv825917864=90, srv1516456189=17, srv869100116=94, srv129442513=8, srv1681976318=26, srv2012670408=52, srv976721895=98, srv637110189=84, srv1798804720=35, srv357102613=70, srv307236757=68, srv750239678=86, srv1377224996=10, srv927085486=95, srv372013614=72, srv791758446=87, srv557505520=79, srv1558056590=21, srv2129336500=64, srv990955342=99, srv1470311891=14, srv636886285=83, srv1461287072=13, srv2023127801=53, srv1834320772=37, srv2075435657=59, srv1679102491=25, srv32966871=69, srv542706559=78, srv1146449825=5, srv1133487801=2, srv1142111283=4, srv2091204350=60, srv178245889=34, srv587124937=81, srv365177051=71} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-02 02:55:36,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-02 02:55:36,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-02 02:55:36,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-02 02:55:36,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-02 02:55:36,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-02 02:55:36,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-02 02:55:36,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-02 02:55:36,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-02 02:55:36,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-02 02:55:36,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-02 02:55:36,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-02 02:55:36,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-02 02:55:36,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-02 02:55:36,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-02 02:55:36,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-02 02:55:36,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-02 02:55:36,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-02 02:55:36,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-02 02:55:36,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-02 02:55:36,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-02 02:55:36,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-02 02:55:36,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-02 02:55:36,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-02 02:55:36,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-02 02:55:36,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-02 02:55:36,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-02 02:55:36,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-02 02:55:36,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-02 02:55:36,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-02 02:55:36,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-02 02:55:36,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-02 02:55:36,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-02 02:55:36,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-02 02:55:36,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-02 02:55:36,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-02 02:55:36,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-02 02:55:36,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-02 02:55:36,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-02 02:55:36,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-02 02:55:36,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-02 02:55:36,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-02 02:55:36,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-02 02:55:36,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-02 02:55:36,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-02 02:55:36,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-02 02:55:36,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-02 02:55:36,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-02 02:55:36,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-02 02:55:36,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-02 02:55:36,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-02 02:55:36,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-02 02:55:36,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-02 02:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-02 02:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-02 02:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-02 02:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-02 02:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-02 02:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-02 02:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-02 02:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-02 02:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-02 02:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-02 02:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-02 02:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-02 02:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-02 02:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-02 02:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-02 02:55:36,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-02 02:55:36,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-02 02:55:36,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-02 02:55:36,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-02 02:55:36,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-02 02:55:36,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-02 02:55:36,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-02 02:55:36,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-02 02:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-02 02:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-02 02:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-02 02:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-02 02:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-02 02:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-02 02:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-02 02:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-02 02:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-02 02:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-02 02:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-02 02:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-02 02:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-02 02:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-02 02:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-02 02:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-02 02:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-02 02:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-02 02:55:36,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-02 02:55:36,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-02 02:55:36,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-02 02:55:36,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-02 02:55:36,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-02 02:55:36,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-02 02:55:36,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-02 02:55:36,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-02 02:55:36,222 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.8071720132083482E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 02:55:36,222 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1485575702:83 , srv1772602161:83 , srv2074077803:83 , srv365177051:83 , srv825917864:83 , srv1062039145:84 , srv122304465:84 , srv129442513:84 , srv1470311891:84 , srv1530097450:84 , srv1537897131:84 , srv1556251875:84 , srv1634408544:84 , srv1663139907:84 , srv1718229461:84 , srv1732224713:84 , srv175288175:84 , srv178245889:84 , srv1834320772:84 , srv1848065831:84 , srv1899729473:84 , srv1900407428:84 , srv1916508018:84 , srv1978370587:84 , srv2000122556:84 , srv2012670408:84 , srv2023127801:84 , srv2055677584:84 , srv2075435657:84 , srv2091204350:84 , srv2104607007:84 , srv2129336500:84 , srv2137657374:84 , srv2145935532:84 , srv307236757:84 , srv372013614:84 , srv579161596:84 , srv626878719:84 , srv636886285:84 , srv637110189:84 , srv750239678:84 , srv791758446:84 , srv818162561:84 , srv851076987:84 , srv857744474:84 , srv860372460:84 , srv976721895:84 , srv1133487801:85 , srv1137871259:85 , srv1142111283:85 , srv1146449825:85 , srv1275365121:85 , srv1317074381:85 , srv1377224996:85 , srv1448932890:85 , srv1461287072:85 , srv150066947:85 , srv1516456189:85 , srv1558056590:85 , srv1672027413:85 , srv1679102491:85 , srv1681976318:85 , srv1714549420:85 , srv1720261349:85 , srv1798804720:85 , srv1799705468:85 , srv1851129693:85 , srv1862358562:85 , srv1876634200:85 , srv1915324676:85 , srv1917035507:85 , srv1990388669:85 , srv2005296181:85 , srv2036614892:85 , srv2039760349:85 , srv2116463846:85 , srv2116759911:85 , srv264585224:85 , srv32966871:85 , srv392707397:85 , srv448176549:85 , srv459990838:85 , srv523179060:85 , srv542706559:85 , srv557505520:85 , srv811639251:85 , srv869100116:85 , srv927085486:85 , srv941332986:85 , srv96927222:85 , srv990955342:85 , srv1058623324:86 , srv1428956008:86 , srv1747144321:86 , srv1968066532:86 , srv206889882:86 , srv357102613:86 , srv396716796:86 , srv587124937:86 , srv733222103:629 }
2023-11-02 02:55:36,250 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=114 (was 114), ProcessCount=169 (was 169), AvailableMemoryMB=11416 (was 11905)
2023-11-02 02:55:36,257 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=114, ProcessCount=169, AvailableMemoryMB=11416
2023-11-02 02:55:36,257 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-02 02:55:36,258 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-11-02 02:55:36,265 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1000878861:84 , srv1095498017:84 , srv1099786859:84 , srv1216453451:84 , srv1217479415:84 , srv1275925291:84 , srv1349740135:84 , srv1508337749:84 , srv1509190963:84 , srv1616421676:84 , srv1647977999:84 , srv1759993342:84 , srv1776575312:84 , srv1851674672:84 , srv2074309029:84 , srv2144110915:84 , srv2144708804:84 , srv351064881:84 , srv377477546:84 , srv394996132:84 , srv449629376:84 , srv504806236:84 , srv528455215:84 , srv528528583:84 , srv700726155:84 , srv752284368:84 , srv922031946:84 , srv94882123:84 , srv961741564:84 , srv1167226273:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-11-02 02:55:36,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-02 02:55:36,266 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1851674672=14, srv1647977999=11, srv961741564=29, srv1508337749=8, srv1000878861=0, srv1099786859=2, srv377477546=19, srv504806236=22, srv1216453451=4, srv752284368=26, srv94882123=28, srv1095498017=1, srv1167226273=3, srv700726155=25, srv1616421676=10, srv528455215=23, srv351064881=18, srv449629376=21, srv1275925291=6, srv922031946=27, srv2074309029=15, srv528528583=24, srv2144110915=16, srv394996132=20, srv1759993342=12, srv1776575312=13, srv1217479415=5, srv1509190963=9, srv1349740135=7, srv2144708804=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-02 02:55:36,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-02 02:55:36,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-02 02:55:36,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-02 02:55:36,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-02 02:55:36,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-02 02:55:36,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-02 02:55:36,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-02 02:55:36,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-02 02:55:36,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-02 02:55:36,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-02 02:55:36,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-02 02:55:36,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-02 02:55:36,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-02 02:55:36,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-02 02:55:36,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-02 02:55:36,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-02 02:55:36,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-02 02:55:36,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-02 02:55:36,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-02 02:55:36,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-02 02:55:36,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-02 02:55:36,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-02 02:55:36,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-02 02:55:36,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-02 02:55:36,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-02 02:55:36,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-02 02:55:36,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-02 02:55:36,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-02 02:55:36,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-02 02:55:36,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-02 02:55:36,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-02 02:55:36,292 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.2116668739070386, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 02:55:36,292 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.99128599986724, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.2116668739070386, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-11-02 02:57:36,280 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5494492 different iterations.  Found a solution that moves 1861 regions; Going from a computed imbalance of 0.99128599986724 to a new imbalance of 4.3436668390529226E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6892592592592592, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 02:57:36,286 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1000878861:90 , srv1095498017:90 , srv1099786859:90 , srv1167226273:90 , srv1216453451:90 , srv1217479415:90 , srv1275925291:90 , srv1349740135:90 , srv1508337749:90 , srv1509190963:90 , srv1616421676:90 , srv1647977999:90 , srv1759993342:90 , srv1776575312:90 , srv1851674672:90 , srv2074309029:90 , srv2144110915:90 , srv2144708804:90 , srv351064881:90 , srv377477546:90 , srv394996132:90 , srv449629376:90 , srv504806236:90 , srv528455215:90 , srv528528583:90 , srv700726155:90 , srv752284368:90 , srv922031946:90 , srv94882123:90 , srv961741564:90 }
2023-11-02 02:57:36,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-02 02:57:36,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1851674672=14, srv1647977999=11, srv961741564=29, srv1508337749=8, srv1000878861=0, srv1099786859=2, srv377477546=19, srv504806236=22, srv1216453451=4, srv752284368=26, srv94882123=28, srv1095498017=1, srv1167226273=3, srv700726155=25, srv1616421676=10, srv528455215=23, srv351064881=18, srv449629376=21, srv1275925291=6, srv922031946=27, srv2074309029=15, srv528528583=24, srv2144110915=16, srv394996132=20, srv1759993342=12, srv1776575312=13, srv1217479415=5, srv1509190963=9, srv1349740135=7, srv2144708804=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-02 02:57:36,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-02 02:57:36,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-02 02:57:36,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-02 02:57:36,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-02 02:57:36,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-02 02:57:36,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-02 02:57:36,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-02 02:57:36,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-02 02:57:36,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-02 02:57:36,304 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-02 02:57:36,304 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1000878861:90 , srv1095498017:90 , srv1099786859:90 , srv1167226273:90 , srv1216453451:90 , srv1217479415:90 , srv1275925291:90 , srv1349740135:90 , srv1508337749:90 , srv1509190963:90 , srv1616421676:90 , srv1647977999:90 , srv1759993342:90 , srv1776575312:90 , srv1851674672:90 , srv2074309029:90 , srv2144110915:90 , srv2144708804:90 , srv351064881:90 , srv377477546:90 , srv394996132:90 , srv449629376:90 , srv504806236:90 , srv528455215:90 , srv528528583:90 , srv700726155:90 , srv752284368:90 , srv922031946:90 , srv94882123:90 , srv961741564:90 }
2023-11-02 02:57:36,317 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=186 (was 114) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 169), AvailableMemoryMB=11397 (was 11416)