2023-11-08 18:59:00,159 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-11-08 18:59:00,437 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-11-08 18:59:00,547 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-11-08 18:59:00,604 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-08 18:59:00,613 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-08 18:59:00,627 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=315, ProcessCount=165, AvailableMemoryMB=7376
2023-11-08 18:59:00,628 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-08 18:59:00,629 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-08 18:59:00,753 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1001370670:84 , srv1005868889:84 , srv1011661180:84 , srv103654390:84 , srv1051331524:84 , srv105137236:84 , srv1055045451:84 , srv1058795086:84 , srv1078136846:84 , srv1095522866:84 , srv1108208942:84 , srv1137057638:84 , srv1140808781:84 , srv1142669503:84 , srv11691851:84 , srv1183795810:84 , srv1184837758:84 , srv1197088648:84 , srv1198685955:84 , srv1222831316:84 , srv1230446791:84 , srv1239112335:84 , srv1262175527:84 , srv1302895708:84 , srv1317108790:84 , srv1371779475:84 , srv1411120011:84 , srv1481985929:84 , srv148335092:84 , srv1493003244:84 , srv1520082618:84 , srv1538698995:84 , srv1541613935:84 , srv1548687662:84 , srv156195583:84 , srv156724900:84 , srv1606319227:84 , srv1623485504:84 , srv1680658456:84 , srv1687795679:84 , srv1693573740:84 , srv1701282972:84 , srv1737114866:84 , srv1738201277:84 , srv1741904313:84 , srv1817999994:84 , srv182370100:84 , srv1824685372:84 , srv1856534359:84 , srv1857129625:84 , srv1904330897:84 , srv1915575139:84 , srv1924940082:84 , srv1951511433:84 , srv1959031823:84 , srv197911122:84 , srv1979182774:84 , srv2031022678:84 , srv2042964498:84 , srv2052945843:84 , srv2071409991:84 , srv208525545:84 , srv2111052061:84 , srv2127074179:84 , srv231061022:84 , srv240024023:84 , srv252015808:84 , srv253015711:84 , srv26527357:84 , srv302463709:84 , srv319857548:84 , srv40437611:84 , srv436060127:84 , srv449384723:84 , srv450157920:84 , srv460431460:84 , srv465624061:84 , srv484844177:84 , srv502217302:84 , srv52293308:84 , srv532124990:84 , srv536900987:84 , srv563295030:84 , srv568095237:84 , srv568221517:84 , srv603549676:84 , srv628479417:84 , srv649183427:84 , srv658855107:84 , srv677276495:84 , srv681632366:84 , srv696071159:84 , srv758834100:84 , srv790214638:84 , srv838187024:84 , srv86279844:84 , srv890373482:84 , srv9685715:84 , srv978816719:84 , srv289138853:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-11-08 18:59:00,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-08 18:59:00,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1481985929=27, srv568095237=84, srv465624061=77, srv26527357=68, srv1693573740=40, srv1317108790=24, srv1137057638=11, srv197911122=55, srv2052945843=59, srv1924940082=52, srv1538698995=31, srv1055045451=6, srv681632366=91, srv52293308=80, srv1262175527=22, srv156195583=34, srv1058795086=7, srv1606319227=36, srv1817999994=45, srv103654390=3, srv1915575139=51, srv677276495=90, srv838187024=95, srv1183795810=15, srv2042964498=58, srv40437611=72, srv252015808=66, srv1142669503=13, srv2031022678=57, srv1680658456=38, srv649183427=88, srv628479417=87, srv1371779475=25, srv1095522866=9, srv182370100=46, srv1005868889=1, srv1979182774=56, srv449384723=74, srv148335092=28, srv1184837758=16, srv1051331524=4, srv1520082618=30, srv568221517=85, srv86279844=96, srv231061022=64, srv1198685955=18, srv1741904313=44, srv1738201277=43, srv696071159=92, srv9685715=98, srv1959031823=54, srv1548687662=33, srv2111052061=62, srv1904330897=50, srv790214638=94, srv436060127=73, srv1302895708=23, srv1230446791=20, srv563295030=83, srv302463709=70, srv1239112335=21, srv890373482=97, srv758834100=93, srv1824685372=47, srv1623485504=37, srv1701282972=41, srv1856534359=48, srv978816719=99, srv319857548=71, srv536900987=82, srv1857129625=49, srv1541613935=32, srv460431460=76, srv1140808781=12, srv1197088648=17, srv502217302=79, srv1411120011=26, srv1951511433=53, srv532124990=81, srv240024023=65, srv603549676=86, srv1078136846=8, srv11691851=14, srv208525545=61, srv484844177=78, srv658855107=89, srv156724900=35, srv1011661180=2, srv1108208942=10, srv2127074179=63, srv450157920=75, srv1001370670=0, srv1222831316=19, srv1493003244=29, srv1687795679=39, srv253015711=67, srv289138853=69, srv1737114866=42, srv105137236=5, srv2071409991=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-08 18:59:00,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-08 18:59:00,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-08 18:59:00,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-08 18:59:00,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-08 18:59:00,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-08 18:59:00,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-08 18:59:00,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-08 18:59:00,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-08 18:59:00,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-08 18:59:00,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-08 18:59:00,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-08 18:59:00,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-08 18:59:00,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-08 18:59:00,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-08 18:59:00,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-08 18:59:00,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-08 18:59:00,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-08 18:59:00,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-08 18:59:00,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-08 18:59:00,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-08 18:59:00,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-08 18:59:00,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-08 18:59:00,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-08 18:59:00,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-08 18:59:00,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-08 18:59:00,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-08 18:59:00,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-08 18:59:00,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-08 18:59:00,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-08 18:59:00,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-08 18:59:00,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-08 18:59:00,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-08 18:59:00,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-08 18:59:00,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-08 18:59:00,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-08 18:59:00,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-08 18:59:00,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-08 18:59:00,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-08 18:59:00,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-08 18:59:00,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-08 18:59:00,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-08 18:59:00,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-08 18:59:00,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-08 18:59:00,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-08 18:59:00,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-08 18:59:00,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-08 18:59:00,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-08 18:59:00,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-08 18:59:00,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-08 18:59:00,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-08 18:59:00,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-08 18:59:00,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-08 18:59:00,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-08 18:59:00,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-08 18:59:00,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-08 18:59:00,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-08 18:59:00,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-08 18:59:00,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-08 18:59:00,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-08 18:59:00,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-08 18:59:00,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-08 18:59:00,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-08 18:59:00,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-08 18:59:00,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-08 18:59:00,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-08 18:59:00,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-08 18:59:00,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-08 18:59:00,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-08 18:59:00,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-08 18:59:00,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-08 18:59:00,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-08 18:59:00,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-08 18:59:00,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-08 18:59:00,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-08 18:59:00,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-08 18:59:00,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-08 18:59:00,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-08 18:59:00,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-08 18:59:00,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-08 18:59:00,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-08 18:59:00,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-08 18:59:00,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-08 18:59:00,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-08 18:59:00,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-08 18:59:00,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-08 18:59:00,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-08 18:59:00,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-08 18:59:00,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-08 18:59:00,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-08 18:59:00,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-08 18:59:00,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-08 18:59:00,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-08 18:59:00,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-08 18:59:00,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-08 18:59:00,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-08 18:59:00,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-08 18:59:00,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-08 18:59:00,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-08 18:59:00,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-08 18:59:00,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-08 18:59:00,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-08 18:59:01,079 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-08 18:59:01,080 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-08 18:59:01,080 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-08 18:59:01,384 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 462 ms to try 5000 different iterations.  Found a solution that moves 4688 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1776832147868034. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040823301714073286, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0503968538023576, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5208888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040823301714073286, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17275, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24108333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-08 18:59:01,399 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1741904313:82 , srv103654390:83 , srv1137057638:83 , srv1262175527:83 , srv1680658456:83 , srv1824685372:83 , srv208525545:83 , srv2111052061:83 , srv681632366:83 , srv1011661180:84 , srv1108208942:84 , srv1687795679:84 , srv1693573740:84 , srv1701282972:84 , srv182370100:84 , srv1856534359:84 , srv2071409991:84 , srv2127074179:84 , srv253015711:84 , srv302463709:84 , srv449384723:84 , srv502217302:84 , srv978816719:84 , srv1183795810:85 , srv1184837758:85 , srv1198685955:85 , srv1230446791:85 , srv1302895708:85 , srv1481985929:85 , srv1904330897:85 , srv1924940082:85 , srv1979182774:85 , srv2031022678:85 , srv2042964498:85 , srv2052945843:85 , srv40437611:85 , srv484844177:85 , srv628479417:85 , srv790214638:85 , srv105137236:86 , srv1055045451:86 , srv1058795086:86 , srv1078136846:86 , srv1140808781:86 , srv1239112335:86 , srv1411120011:86 , srv1520082618:86 , srv156195583:86 , srv1606319227:86 , srv1959031823:86 , srv197911122:86 , srv231061022:86 , srv240024023:86 , srv252015808:86 , srv436060127:86 , srv532124990:86 , srv563295030:86 , srv568095237:86 , srv838187024:86 , srv1001370670:87 , srv1197088648:87 , srv1371779475:87 , srv148335092:87 , srv1541613935:87 , srv1548687662:87 , srv1738201277:87 , srv1915575139:87 , srv319857548:87 , srv450157920:87 , srv460431460:87 , srv52293308:87 , srv568221517:87 , srv603549676:87 , srv9685715:87 , srv1051331524:88 , srv1142669503:88 , srv1493003244:88 , srv156724900:88 , srv26527357:88 , srv649183427:88 , srv11691851:89 , srv1623485504:89 , srv1737114866:89 , srv1857129625:89 , srv536900987:89 , srv658855107:89 , srv677276495:89 , srv758834100:89 , srv1005868889:90 , srv1095522866:90 , srv1222831316:90 , srv1951511433:90 , srv696071159:90 , srv86279844:90 , srv1817999994:91 , srv465624061:91 , srv1317108790:92 , srv1538698995:93 , srv890373482:93 , srv289138853:453 }
2023-11-08 18:59:01,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-08 18:59:01,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1481985929=27, srv568095237=84, srv465624061=77, srv26527357=68, srv1693573740=40, srv1317108790=24, srv1137057638=11, srv197911122=55, srv2052945843=59, srv1924940082=52, srv1538698995=31, srv1055045451=6, srv681632366=91, srv52293308=80, srv1262175527=22, srv156195583=34, srv1058795086=7, srv1606319227=36, srv1817999994=45, srv103654390=3, srv1915575139=51, srv677276495=90, srv838187024=95, srv1183795810=15, srv2042964498=58, srv40437611=72, srv252015808=66, srv1142669503=13, srv2031022678=57, srv1680658456=38, srv649183427=88, srv628479417=87, srv1371779475=25, srv1095522866=9, srv182370100=46, srv1005868889=1, srv1979182774=56, srv449384723=74, srv148335092=28, srv1184837758=16, srv1051331524=4, srv1520082618=30, srv568221517=85, srv86279844=96, srv231061022=64, srv1198685955=18, srv1741904313=44, srv1738201277=43, srv696071159=92, srv9685715=98, srv1959031823=54, srv1548687662=33, srv2111052061=62, srv1904330897=50, srv790214638=94, srv436060127=73, srv1302895708=23, srv1230446791=20, srv563295030=83, srv302463709=70, srv1239112335=21, srv890373482=97, srv758834100=93, srv1824685372=47, srv1623485504=37, srv1701282972=41, srv1856534359=48, srv978816719=99, srv319857548=71, srv536900987=82, srv1857129625=49, srv1541613935=32, srv460431460=76, srv1140808781=12, srv1197088648=17, srv502217302=79, srv1411120011=26, srv1951511433=53, srv532124990=81, srv240024023=65, srv603549676=86, srv1078136846=8, srv11691851=14, srv208525545=61, srv484844177=78, srv658855107=89, srv156724900=35, srv1011661180=2, srv1108208942=10, srv2127074179=63, srv450157920=75, srv1001370670=0, srv1222831316=19, srv1493003244=29, srv1687795679=39, srv253015711=67, srv289138853=69, srv1737114866=42, srv105137236=5, srv2071409991=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-08 18:59:01,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-08 18:59:01,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-08 18:59:01,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-08 18:59:01,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-08 18:59:01,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-08 18:59:01,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-08 18:59:01,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-08 18:59:01,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-08 18:59:01,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-08 18:59:01,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-08 18:59:01,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-08 18:59:01,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-08 18:59:01,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-08 18:59:01,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-08 18:59:01,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-08 18:59:01,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-08 18:59:01,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-08 18:59:01,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-08 18:59:01,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-08 18:59:01,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-08 18:59:01,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-08 18:59:01,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-08 18:59:01,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-08 18:59:01,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-08 18:59:01,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-08 18:59:01,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-08 18:59:01,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-08 18:59:01,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-08 18:59:01,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-08 18:59:01,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-08 18:59:01,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-08 18:59:01,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-08 18:59:01,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-08 18:59:01,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-08 18:59:01,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-08 18:59:01,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-08 18:59:01,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-08 18:59:01,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-08 18:59:01,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-08 18:59:01,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-08 18:59:01,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-08 18:59:01,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-08 18:59:01,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-08 18:59:01,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-08 18:59:01,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-08 18:59:01,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-08 18:59:01,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-08 18:59:01,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-08 18:59:01,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-08 18:59:01,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-08 18:59:01,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-08 18:59:01,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-08 18:59:01,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-08 18:59:01,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-08 18:59:01,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-08 18:59:01,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-08 18:59:01,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-08 18:59:01,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-08 18:59:01,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-08 18:59:01,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-08 18:59:01,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-08 18:59:01,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-08 18:59:01,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-08 18:59:01,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-08 18:59:01,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-08 18:59:01,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-08 18:59:01,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-08 18:59:01,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-08 18:59:01,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-08 18:59:01,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-08 18:59:01,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-08 18:59:01,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-08 18:59:01,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-08 18:59:01,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-08 18:59:01,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-08 18:59:01,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-08 18:59:01,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-08 18:59:01,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-08 18:59:01,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-08 18:59:01,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-08 18:59:01,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-08 18:59:01,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-08 18:59:01,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-08 18:59:01,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-08 18:59:01,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-08 18:59:01,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-08 18:59:01,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-08 18:59:01,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-08 18:59:01,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-08 18:59:01,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-08 18:59:01,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-08 18:59:01,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-08 18:59:01,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-08 18:59:01,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-08 18:59:01,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-08 18:59:01,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-08 18:59:01,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-08 18:59:01,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-08 18:59:01,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-08 18:59:01,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-08 18:59:01,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-08 18:59:01,598 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040823301714073286, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0503968538023576, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040823301714073286, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17275, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24108333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-08 18:59:01,598 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-08 18:59:01,599 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17765038870919758, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040823301714073286, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0503968538023576, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040823301714073286, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17275, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24108333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-08 18:59:01,785 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 320 ms to try 5000 different iterations.  Found a solution that moves 2633 regions; Going from a computed imbalance of 0.17765038870919758 to a new imbalance of 0.014589087761531642. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0227397646178379); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030930209989769277, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29255555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0227397646178379); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04575, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-08 18:59:01,793 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1538698995:79 , srv890373482:79 , srv86279844:80 , srv11691851:81 , srv1317108790:81 , srv156724900:81 , srv1817999994:81 , srv1951511433:81 , srv26527357:81 , srv1005868889:82 , srv1051331524:82 , srv1095522866:82 , srv148335092:82 , srv1737114866:82 , srv465624061:82 , srv1001370670:83 , srv1493003244:83 , srv1520082618:83 , srv1541613935:83 , srv252015808:83 , srv658855107:83 , srv696071159:83 , srv1058795086:84 , srv1857129625:84 , srv450157920:84 , srv568221517:84 , srv603549676:84 , srv649183427:84 , srv758834100:84 , srv838187024:84 , srv1078136846:85 , srv1142669503:85 , srv1183795810:85 , srv1222831316:85 , srv1239112335:85 , srv1481985929:85 , srv156195583:85 , srv1623485504:85 , srv1738201277:85 , srv1915575139:85 , srv319857548:85 , srv52293308:85 , srv9685715:85 , srv105137236:86 , srv1140808781:86 , srv1197088648:86 , srv1548687662:86 , srv1687795679:86 , srv1856534359:86 , srv1924940082:86 , srv2052945843:86 , srv231061022:86 , srv240024023:86 , srv302463709:86 , srv436060127:86 , srv532124990:86 , srv563295030:86 , srv628479417:86 , srv677276495:86 , srv1108208942:87 , srv1371779475:87 , srv1411120011:87 , srv1680658456:87 , srv1701282972:87 , srv1904330897:87 , srv1959031823:87 , srv1979182774:87 , srv2071409991:87 , srv2127074179:87 , srv40437611:87 , srv449384723:87 , srv460431460:87 , srv790214638:87 , srv1011661180:88 , srv1198685955:88 , srv1230446791:88 , srv1302895708:88 , srv1693573740:88 , srv1741904313:88 , srv197911122:88 , srv2031022678:88 , srv2042964498:88 , srv253015711:88 , srv484844177:88 , srv568095237:88 , srv1055045451:89 , srv1137057638:89 , srv1184837758:89 , srv1262175527:89 , srv182370100:89 , srv1824685372:89 , srv502217302:89 , srv536900987:89 , srv978816719:89 , srv1606319227:90 , srv208525545:90 , srv2111052061:90 , srv103654390:91 , srv681632366:91 , srv289138853:523 }
2023-11-08 18:59:01,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-08 18:59:01,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1481985929=27, srv568095237=84, srv465624061=77, srv26527357=68, srv1693573740=40, srv1317108790=24, srv1137057638=11, srv197911122=55, srv2052945843=59, srv1924940082=52, srv1538698995=31, srv1055045451=6, srv681632366=91, srv52293308=80, srv1262175527=22, srv156195583=34, srv1058795086=7, srv1606319227=36, srv1817999994=45, srv103654390=3, srv1915575139=51, srv677276495=90, srv838187024=95, srv1183795810=15, srv2042964498=58, srv40437611=72, srv252015808=66, srv1142669503=13, srv2031022678=57, srv1680658456=38, srv649183427=88, srv628479417=87, srv1371779475=25, srv1095522866=9, srv182370100=46, srv1005868889=1, srv1979182774=56, srv449384723=74, srv148335092=28, srv1184837758=16, srv1051331524=4, srv1520082618=30, srv568221517=85, srv86279844=96, srv231061022=64, srv1198685955=18, srv1741904313=44, srv1738201277=43, srv696071159=92, srv9685715=98, srv1959031823=54, srv1548687662=33, srv2111052061=62, srv1904330897=50, srv790214638=94, srv436060127=73, srv1302895708=23, srv1230446791=20, srv563295030=83, srv302463709=70, srv1239112335=21, srv890373482=97, srv758834100=93, srv1824685372=47, srv1623485504=37, srv1701282972=41, srv1856534359=48, srv978816719=99, srv319857548=71, srv536900987=82, srv1857129625=49, srv1541613935=32, srv460431460=76, srv1140808781=12, srv1197088648=17, srv502217302=79, srv1411120011=26, srv1951511433=53, srv532124990=81, srv240024023=65, srv603549676=86, srv1078136846=8, srv11691851=14, srv208525545=61, srv484844177=78, srv658855107=89, srv156724900=35, srv1011661180=2, srv1108208942=10, srv2127074179=63, srv450157920=75, srv1001370670=0, srv1222831316=19, srv1493003244=29, srv1687795679=39, srv253015711=67, srv289138853=69, srv1737114866=42, srv105137236=5, srv2071409991=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-08 18:59:01,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-08 18:59:01,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-08 18:59:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-08 18:59:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-08 18:59:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-08 18:59:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-08 18:59:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-08 18:59:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-08 18:59:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-08 18:59:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-08 18:59:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-08 18:59:01,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-08 18:59:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-08 18:59:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-08 18:59:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-08 18:59:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-08 18:59:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-08 18:59:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-08 18:59:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-08 18:59:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-08 18:59:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-08 18:59:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-08 18:59:01,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-08 18:59:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-08 18:59:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-08 18:59:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-08 18:59:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-08 18:59:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-08 18:59:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-08 18:59:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-08 18:59:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-08 18:59:01,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-08 18:59:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-08 18:59:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-08 18:59:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-08 18:59:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-08 18:59:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-08 18:59:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-08 18:59:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-08 18:59:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-08 18:59:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-08 18:59:01,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-08 18:59:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-08 18:59:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-08 18:59:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-08 18:59:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-08 18:59:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-08 18:59:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-08 18:59:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-08 18:59:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-08 18:59:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-08 18:59:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-08 18:59:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-08 18:59:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-08 18:59:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-08 18:59:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-08 18:59:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-08 18:59:01,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-08 18:59:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-08 18:59:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-08 18:59:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-08 18:59:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-08 18:59:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-08 18:59:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-08 18:59:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-08 18:59:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-08 18:59:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-08 18:59:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-08 18:59:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-08 18:59:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-08 18:59:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-08 18:59:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-08 18:59:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-08 18:59:01,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-08 18:59:01,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-08 18:59:01,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-08 18:59:01,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-08 18:59:01,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-08 18:59:01,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-08 18:59:01,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-08 18:59:01,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-08 18:59:01,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-08 18:59:01,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-08 18:59:01,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-08 18:59:01,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-08 18:59:01,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-08 18:59:01,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-08 18:59:01,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-08 18:59:01,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-08 18:59:01,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-08 18:59:01,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-08 18:59:01,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-08 18:59:01,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-08 18:59:01,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-08 18:59:01,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-08 18:59:01,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-08 18:59:01,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-08 18:59:01,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-08 18:59:01,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-08 18:59:01,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-08 18:59:01,815 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-08 18:59:01,900 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0227397646178379); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030930209989769277, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0227397646178379); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04575, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-08 18:59:01,901 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-08 18:59:01,901 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014570651101477006, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0227397646178379); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030930209989769277, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0227397646178379); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04575, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-08 18:59:02,134 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 309 ms to try 5000 different iterations.  Found a solution that moves 954 regions; Going from a computed imbalance of 0.014570651101477006 to a new imbalance of 5.75387947309767E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006434413029274749); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006122578771996662); MoveCostFunction : (multiplier=7.0, imbalance=0.106, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006434413029274749); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-08 18:59:02,135 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv536900987:81 , srv1606319227:82 , srv677276495:82 , srv1184837758:83 , srv1222831316:83 , srv1371779475:83 , srv197911122:83 , srv568095237:83 , srv758834100:83 , srv103654390:84 , srv1055045451:84 , srv1142669503:84 , srv1198685955:84 , srv1230446791:84 , srv1302895708:84 , srv1623485504:84 , srv1959031823:84 , srv2031022678:84 , srv2042964498:84 , srv460431460:84 , srv502217302:84 , srv681632366:84 , srv696071159:84 , srv978816719:84 , srv1005868889:85 , srv1011661180:85 , srv1137057638:85 , srv1197088648:85 , srv1262175527:85 , srv1317108790:85 , srv1411120011:85 , srv1538698995:85 , srv1548687662:85 , srv1693573740:85 , srv1738201277:85 , srv1824685372:85 , srv1857129625:85 , srv1915575139:85 , srv1979182774:85 , srv208525545:85 , srv2111052061:85 , srv231061022:85 , srv240024023:85 , srv319857548:85 , srv436060127:85 , srv465624061:85 , srv484844177:85 , srv52293308:85 , srv532124990:85 , srv563295030:85 , srv649183427:85 , srv658855107:85 , srv105137236:86 , srv1078136846:86 , srv1108208942:86 , srv1140808781:86 , srv1239112335:86 , srv1493003244:86 , srv156195583:86 , srv1817999994:86 , srv182370100:86 , srv1904330897:86 , srv1924940082:86 , srv1951511433:86 , srv2127074179:86 , srv253015711:86 , srv40437611:86 , srv449384723:86 , srv450157920:86 , srv568221517:86 , srv603549676:86 , srv628479417:86 , srv790214638:86 , srv838187024:86 , srv890373482:86 , srv9685715:86 , srv1051331524:87 , srv1058795086:87 , srv1095522866:87 , srv1481985929:87 , srv1680658456:87 , srv1687795679:87 , srv1701282972:87 , srv1737114866:87 , srv1741904313:87 , srv2052945843:87 , srv2071409991:87 , srv302463709:87 , srv86279844:87 , srv1001370670:88 , srv11691851:88 , srv1183795810:88 , srv148335092:88 , srv1541613935:88 , srv156724900:88 , srv1856534359:88 , srv26527357:88 , srv252015808:89 , srv1520082618:90 , srv289138853:539 }
2023-11-08 18:59:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-08 18:59:02,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1481985929=27, srv568095237=84, srv465624061=77, srv26527357=68, srv1693573740=40, srv1317108790=24, srv1137057638=11, srv197911122=55, srv2052945843=59, srv1924940082=52, srv1538698995=31, srv1055045451=6, srv681632366=91, srv52293308=80, srv1262175527=22, srv156195583=34, srv1058795086=7, srv1606319227=36, srv1817999994=45, srv103654390=3, srv1915575139=51, srv677276495=90, srv838187024=95, srv1183795810=15, srv2042964498=58, srv40437611=72, srv252015808=66, srv1142669503=13, srv2031022678=57, srv1680658456=38, srv649183427=88, srv628479417=87, srv1371779475=25, srv1095522866=9, srv182370100=46, srv1005868889=1, srv1979182774=56, srv449384723=74, srv148335092=28, srv1184837758=16, srv1051331524=4, srv1520082618=30, srv568221517=85, srv86279844=96, srv231061022=64, srv1198685955=18, srv1741904313=44, srv1738201277=43, srv696071159=92, srv9685715=98, srv1959031823=54, srv1548687662=33, srv2111052061=62, srv1904330897=50, srv790214638=94, srv436060127=73, srv1302895708=23, srv1230446791=20, srv563295030=83, srv302463709=70, srv1239112335=21, srv890373482=97, srv758834100=93, srv1824685372=47, srv1623485504=37, srv1701282972=41, srv1856534359=48, srv978816719=99, srv319857548=71, srv536900987=82, srv1857129625=49, srv1541613935=32, srv460431460=76, srv1140808781=12, srv1197088648=17, srv502217302=79, srv1411120011=26, srv1951511433=53, srv532124990=81, srv240024023=65, srv603549676=86, srv1078136846=8, srv11691851=14, srv208525545=61, srv484844177=78, srv658855107=89, srv156724900=35, srv1011661180=2, srv1108208942=10, srv2127074179=63, srv450157920=75, srv1001370670=0, srv1222831316=19, srv1493003244=29, srv1687795679=39, srv253015711=67, srv289138853=69, srv1737114866=42, srv105137236=5, srv2071409991=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-08 18:59:02,154 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-08 18:59:02,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-08 18:59:02,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-08 18:59:02,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-08 18:59:02,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-08 18:59:02,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-08 18:59:02,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-08 18:59:02,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-08 18:59:02,155 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-08 18:59:02,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-08 18:59:02,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-08 18:59:02,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-08 18:59:02,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-08 18:59:02,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-08 18:59:02,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-08 18:59:02,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-08 18:59:02,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-08 18:59:02,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-08 18:59:02,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-08 18:59:02,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-08 18:59:02,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-08 18:59:02,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-08 18:59:02,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-08 18:59:02,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-08 18:59:02,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-08 18:59:02,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-08 18:59:02,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-08 18:59:02,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-08 18:59:02,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-08 18:59:02,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-08 18:59:02,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-08 18:59:02,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-08 18:59:02,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-08 18:59:02,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-08 18:59:02,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-08 18:59:02,158 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-08 18:59:02,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-08 18:59:02,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-08 18:59:02,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-08 18:59:02,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-08 18:59:02,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-08 18:59:02,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-08 18:59:02,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-08 18:59:02,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-08 18:59:02,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-08 18:59:02,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-08 18:59:02,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-08 18:59:02,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-08 18:59:02,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-08 18:59:02,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-08 18:59:02,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-08 18:59:02,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-08 18:59:02,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-08 18:59:02,160 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-08 18:59:02,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-08 18:59:02,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-08 18:59:02,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-08 18:59:02,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-08 18:59:02,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-08 18:59:02,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-08 18:59:02,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-08 18:59:02,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-08 18:59:02,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-08 18:59:02,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-08 18:59:02,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-08 18:59:02,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-08 18:59:02,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-08 18:59:02,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-08 18:59:02,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-08 18:59:02,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-08 18:59:02,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-08 18:59:02,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-08 18:59:02,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-08 18:59:02,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-08 18:59:02,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-08 18:59:02,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-08 18:59:02,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-08 18:59:02,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-08 18:59:02,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-08 18:59:02,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-08 18:59:02,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-08 18:59:02,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-08 18:59:02,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-08 18:59:02,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-08 18:59:02,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-08 18:59:02,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-08 18:59:02,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-08 18:59:02,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-08 18:59:02,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-08 18:59:02,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-08 18:59:02,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-08 18:59:02,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-08 18:59:02,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-08 18:59:02,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-08 18:59:02,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-08 18:59:02,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-08 18:59:02,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-08 18:59:02,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-08 18:59:02,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-08 18:59:02,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-08 18:59:02,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-08 18:59:02,265 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006434413029274749); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006122578771996662); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006434413029274749); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-08 18:59:02,265 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-08 18:59:02,265 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.687078965341789E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006434413029274749); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006122578771996662); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006434413029274749); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-08 18:59:02,429 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 258 ms to try 5000 different iterations.  Found a solution that moves 218 regions; Going from a computed imbalance of 5.687078965341789E-4 to a new imbalance of 1.7090324307673737E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=2.735366466207394E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.02422222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=2.735366466207394E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-08 18:59:02,430 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1055045451:83 , srv1095522866:83 , srv1520082618:83 , srv182370100:83 , srv536900987:83 , srv1001370670:84 , srv103654390:84 , srv105137236:84 , srv1140808781:84 , srv11691851:84 , srv1183795810:84 , srv1197088648:84 , srv1222831316:84 , srv1317108790:84 , srv1411120011:84 , srv1538698995:84 , srv1541613935:84 , srv1548687662:84 , srv1606319227:84 , srv1623485504:84 , srv1701282972:84 , srv1737114866:84 , srv1817999994:84 , srv1856534359:84 , srv1857129625:84 , srv1904330897:84 , srv2052945843:84 , srv2071409991:84 , srv208525545:84 , srv2111052061:84 , srv253015711:84 , srv319857548:84 , srv40437611:84 , srv460431460:84 , srv465624061:84 , srv484844177:84 , srv658855107:84 , srv681632366:84 , srv790214638:84 , srv890373482:84 , srv9685715:84 , srv1005868889:85 , srv1011661180:85 , srv1058795086:85 , srv1108208942:85 , srv1137057638:85 , srv1142669503:85 , srv1184837758:85 , srv1198685955:85 , srv1230446791:85 , srv1239112335:85 , srv1262175527:85 , srv1302895708:85 , srv1371779475:85 , srv1481985929:85 , srv148335092:85 , srv1493003244:85 , srv156195583:85 , srv156724900:85 , srv1680658456:85 , srv1687795679:85 , srv1693573740:85 , srv1738201277:85 , srv1741904313:85 , srv1824685372:85 , srv1915575139:85 , srv1924940082:85 , srv1951511433:85 , srv1959031823:85 , srv197911122:85 , srv1979182774:85 , srv2031022678:85 , srv2042964498:85 , srv2127074179:85 , srv231061022:85 , srv240024023:85 , srv252015808:85 , srv26527357:85 , srv302463709:85 , srv436060127:85 , srv449384723:85 , srv450157920:85 , srv502217302:85 , srv52293308:85 , srv532124990:85 , srv563295030:85 , srv568095237:85 , srv568221517:85 , srv603549676:85 , srv628479417:85 , srv649183427:85 , srv677276495:85 , srv696071159:85 , srv86279844:85 , srv978816719:85 , srv1051331524:86 , srv1078136846:86 , srv758834100:86 , srv838187024:86 , srv289138853:627 }
2023-11-08 18:59:02,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-08 18:59:02,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1481985929=27, srv568095237=84, srv465624061=77, srv26527357=68, srv1693573740=40, srv1317108790=24, srv1137057638=11, srv197911122=55, srv2052945843=59, srv1924940082=52, srv1538698995=31, srv1055045451=6, srv681632366=91, srv52293308=80, srv1262175527=22, srv156195583=34, srv1058795086=7, srv1606319227=36, srv1817999994=45, srv103654390=3, srv1915575139=51, srv677276495=90, srv838187024=95, srv1183795810=15, srv2042964498=58, srv40437611=72, srv252015808=66, srv1142669503=13, srv2031022678=57, srv1680658456=38, srv649183427=88, srv628479417=87, srv1371779475=25, srv1095522866=9, srv182370100=46, srv1005868889=1, srv1979182774=56, srv449384723=74, srv148335092=28, srv1184837758=16, srv1051331524=4, srv1520082618=30, srv568221517=85, srv86279844=96, srv231061022=64, srv1198685955=18, srv1741904313=44, srv1738201277=43, srv696071159=92, srv9685715=98, srv1959031823=54, srv1548687662=33, srv2111052061=62, srv1904330897=50, srv790214638=94, srv436060127=73, srv1302895708=23, srv1230446791=20, srv563295030=83, srv302463709=70, srv1239112335=21, srv890373482=97, srv758834100=93, srv1824685372=47, srv1623485504=37, srv1701282972=41, srv1856534359=48, srv978816719=99, srv319857548=71, srv536900987=82, srv1857129625=49, srv1541613935=32, srv460431460=76, srv1140808781=12, srv1197088648=17, srv502217302=79, srv1411120011=26, srv1951511433=53, srv532124990=81, srv240024023=65, srv603549676=86, srv1078136846=8, srv11691851=14, srv208525545=61, srv484844177=78, srv658855107=89, srv156724900=35, srv1011661180=2, srv1108208942=10, srv2127074179=63, srv450157920=75, srv1001370670=0, srv1222831316=19, srv1493003244=29, srv1687795679=39, srv253015711=67, srv289138853=69, srv1737114866=42, srv105137236=5, srv2071409991=60} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-08 18:59:02,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-08 18:59:02,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-08 18:59:02,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-08 18:59:02,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-08 18:59:02,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-08 18:59:02,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-08 18:59:02,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-08 18:59:02,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-08 18:59:02,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-08 18:59:02,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-08 18:59:02,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-08 18:59:02,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-08 18:59:02,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-08 18:59:02,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-08 18:59:02,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-08 18:59:02,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-08 18:59:02,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-08 18:59:02,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-08 18:59:02,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-08 18:59:02,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-08 18:59:02,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-08 18:59:02,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-08 18:59:02,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-08 18:59:02,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-08 18:59:02,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-08 18:59:02,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-08 18:59:02,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-08 18:59:02,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-08 18:59:02,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-08 18:59:02,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-08 18:59:02,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-08 18:59:02,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-08 18:59:02,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-08 18:59:02,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-08 18:59:02,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-08 18:59:02,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-08 18:59:02,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-08 18:59:02,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-08 18:59:02,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-08 18:59:02,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-08 18:59:02,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-08 18:59:02,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-08 18:59:02,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-08 18:59:02,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-08 18:59:02,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-08 18:59:02,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-08 18:59:02,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-08 18:59:02,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-08 18:59:02,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-08 18:59:02,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-08 18:59:02,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-08 18:59:02,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-08 18:59:02,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-08 18:59:02,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-08 18:59:02,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-08 18:59:02,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-08 18:59:02,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-08 18:59:02,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-08 18:59:02,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-08 18:59:02,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-08 18:59:02,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-08 18:59:02,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-08 18:59:02,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-08 18:59:02,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-08 18:59:02,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-08 18:59:02,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-08 18:59:02,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-08 18:59:02,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-08 18:59:02,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-08 18:59:02,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-08 18:59:02,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-08 18:59:02,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-08 18:59:02,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-08 18:59:02,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-08 18:59:02,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-08 18:59:02,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-08 18:59:02,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-08 18:59:02,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-08 18:59:02,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-08 18:59:02,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-08 18:59:02,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-08 18:59:02,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-08 18:59:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-08 18:59:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-08 18:59:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-08 18:59:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-08 18:59:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-08 18:59:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-08 18:59:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-08 18:59:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-08 18:59:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-08 18:59:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-08 18:59:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-08 18:59:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-08 18:59:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-08 18:59:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-08 18:59:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-08 18:59:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-08 18:59:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-08 18:59:02,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-08 18:59:02,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-08 18:59:02,544 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.5563855681805597E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=2.735366466207394E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=2.735366466207394E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-08 18:59:02,544 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1055045451:83 , srv1095522866:83 , srv1520082618:83 , srv182370100:83 , srv536900987:83 , srv1001370670:84 , srv103654390:84 , srv105137236:84 , srv1140808781:84 , srv11691851:84 , srv1183795810:84 , srv1197088648:84 , srv1222831316:84 , srv1317108790:84 , srv1411120011:84 , srv1538698995:84 , srv1541613935:84 , srv1548687662:84 , srv1606319227:84 , srv1623485504:84 , srv1701282972:84 , srv1737114866:84 , srv1817999994:84 , srv1856534359:84 , srv1857129625:84 , srv1904330897:84 , srv2052945843:84 , srv2071409991:84 , srv208525545:84 , srv2111052061:84 , srv253015711:84 , srv319857548:84 , srv40437611:84 , srv460431460:84 , srv465624061:84 , srv484844177:84 , srv658855107:84 , srv681632366:84 , srv790214638:84 , srv890373482:84 , srv9685715:84 , srv1005868889:85 , srv1011661180:85 , srv1058795086:85 , srv1108208942:85 , srv1137057638:85 , srv1142669503:85 , srv1184837758:85 , srv1198685955:85 , srv1230446791:85 , srv1239112335:85 , srv1262175527:85 , srv1302895708:85 , srv1371779475:85 , srv1481985929:85 , srv148335092:85 , srv1493003244:85 , srv156195583:85 , srv156724900:85 , srv1680658456:85 , srv1687795679:85 , srv1693573740:85 , srv1738201277:85 , srv1741904313:85 , srv1824685372:85 , srv1915575139:85 , srv1924940082:85 , srv1951511433:85 , srv1959031823:85 , srv197911122:85 , srv1979182774:85 , srv2031022678:85 , srv2042964498:85 , srv2127074179:85 , srv231061022:85 , srv240024023:85 , srv252015808:85 , srv26527357:85 , srv302463709:85 , srv436060127:85 , srv449384723:85 , srv450157920:85 , srv502217302:85 , srv52293308:85 , srv532124990:85 , srv563295030:85 , srv568095237:85 , srv568221517:85 , srv603549676:85 , srv628479417:85 , srv649183427:85 , srv677276495:85 , srv696071159:85 , srv86279844:85 , srv978816719:85 , srv1051331524:86 , srv1078136846:86 , srv758834100:86 , srv838187024:86 , srv289138853:627 }
2023-11-08 18:59:02,571 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=315 (was 315), ProcessCount=165 (was 165), AvailableMemoryMB=6841 (was 7376)
2023-11-08 18:59:02,578 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=315, ProcessCount=165, AvailableMemoryMB=6840
2023-11-08 18:59:02,578 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-08 18:59:02,579 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-11-08 18:59:02,586 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv100731255:84 , srv1047071139:84 , srv10792002:84 , srv1093311316:84 , srv1134122664:84 , srv1206757733:84 , srv1207109090:84 , srv122296691:84 , srv1371786278:84 , srv1445869306:84 , srv1502327631:84 , srv1546864417:84 , srv1583375807:84 , srv16694316:84 , srv166973741:84 , srv167751652:84 , srv1729732830:84 , srv1823381647:84 , srv1838612774:84 , srv1928409924:84 , srv284947390:84 , srv397886913:84 , srv490129263:84 , srv537712402:84 , srv677279250:84 , srv797588086:84 , srv800217944:84 , srv882114207:84 , srv912221729:84 , srv471504642:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-11-08 18:59:02,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-08 18:59:02,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1729732830=16, srv537712402=24, srv677279250=25, srv1093311316=3, srv1928409924=19, srv1134122664=4, srv16694316=13, srv1583375807=12, srv912221729=29, srv490129263=23, srv1546864417=11, srv1206757733=5, srv100731255=0, srv167751652=15, srv166973741=14, srv1502327631=10, srv471504642=22, srv1445869306=9, srv1823381647=17, srv1047071139=1, srv882114207=28, srv122296691=7, srv397886913=21, srv10792002=2, srv1838612774=18, srv1207109090=6, srv284947390=20, srv797588086=26, srv800217944=27, srv1371786278=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-08 18:59:02,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-08 18:59:02,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-08 18:59:02,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-08 18:59:02,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-08 18:59:02,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-08 18:59:02,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-08 18:59:02,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-08 18:59:02,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-08 18:59:02,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-08 18:59:02,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-08 18:59:02,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-08 18:59:02,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-08 18:59:02,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-08 18:59:02,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-08 18:59:02,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-08 18:59:02,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-08 18:59:02,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-08 18:59:02,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-08 18:59:02,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-08 18:59:02,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-08 18:59:02,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-08 18:59:02,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-08 18:59:02,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-08 18:59:02,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-08 18:59:02,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-08 18:59:02,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-08 18:59:02,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-08 18:59:02,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-08 18:59:02,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-08 18:59:02,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-08 18:59:02,598 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-08 18:59:02,623 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0877629742178119, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-08 18:59:02,624 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912469581530315, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0877629742178119, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-11-08 19:01:02,602 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9059866 different iterations.  Found a solution that moves 1832 regions; Going from a computed imbalance of 0.9912469581530315 to a new imbalance of 4.275979392340115E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6785185185185185, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-08 19:01:02,608 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv100731255:90 , srv1047071139:90 , srv10792002:90 , srv1093311316:90 , srv1134122664:90 , srv1206757733:90 , srv1207109090:90 , srv122296691:90 , srv1371786278:90 , srv1445869306:90 , srv1502327631:90 , srv1546864417:90 , srv1583375807:90 , srv16694316:90 , srv166973741:90 , srv167751652:90 , srv1729732830:90 , srv1823381647:90 , srv1838612774:90 , srv1928409924:90 , srv284947390:90 , srv397886913:90 , srv471504642:90 , srv490129263:90 , srv537712402:90 , srv677279250:90 , srv797588086:90 , srv800217944:90 , srv882114207:90 , srv912221729:90 }
2023-11-08 19:01:02,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-08 19:01:02,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1729732830=16, srv537712402=24, srv677279250=25, srv1093311316=3, srv1928409924=19, srv1134122664=4, srv16694316=13, srv1583375807=12, srv912221729=29, srv490129263=23, srv1546864417=11, srv1206757733=5, srv100731255=0, srv167751652=15, srv166973741=14, srv1502327631=10, srv471504642=22, srv1445869306=9, srv1823381647=17, srv1047071139=1, srv882114207=28, srv122296691=7, srv397886913=21, srv10792002=2, srv1838612774=18, srv1207109090=6, srv284947390=20, srv797588086=26, srv800217944=27, srv1371786278=8} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-08 19:01:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-08 19:01:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-08 19:01:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-08 19:01:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-08 19:01:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-08 19:01:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-08 19:01:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-08 19:01:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-08 19:01:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-08 19:01:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-08 19:01:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-08 19:01:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-08 19:01:02,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-08 19:01:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-08 19:01:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-08 19:01:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-08 19:01:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-08 19:01:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-08 19:01:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-08 19:01:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-08 19:01:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-08 19:01:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-08 19:01:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-08 19:01:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-08 19:01:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-08 19:01:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-08 19:01:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-08 19:01:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-08 19:01:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-08 19:01:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-08 19:01:02,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-08 19:01:02,623 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-08 19:01:02,623 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv100731255:90 , srv1047071139:90 , srv10792002:90 , srv1093311316:90 , srv1134122664:90 , srv1206757733:90 , srv1207109090:90 , srv122296691:90 , srv1371786278:90 , srv1445869306:90 , srv1502327631:90 , srv1546864417:90 , srv1583375807:90 , srv16694316:90 , srv166973741:90 , srv167751652:90 , srv1729732830:90 , srv1823381647:90 , srv1838612774:90 , srv1928409924:90 , srv284947390:90 , srv397886913:90 , srv471504642:90 , srv490129263:90 , srv537712402:90 , srv677279250:90 , srv797588086:90 , srv800217944:90 , srv882114207:90 , srv912221729:90 }
2023-11-08 19:01:02,634 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=213 (was 315), ProcessCount=168 (was 165) - ProcessCount LEAK? -, AvailableMemoryMB=6606 (was 6840)