2024-04-03 03:53:17,739 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-04-03 03:53:18,094 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-04-03 03:53:18,216 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-04-03 03:53:18,279 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-04-03 03:53:18,286 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-04-03 03:53:18,305 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=300, ProcessCount=170, AvailableMemoryMB=11251
2024-04-03 03:53:18,307 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-04-03 03:53:18,307 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-04-03 03:53:18,447 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1004456083:84 , srv1013659809:84 , srv1018386950:84 , srv1054876834:84 , srv1055719027:84 , srv1063313261:84 , srv1079283912:84 , srv1083781996:84 , srv1138758271:84 , srv11391989:84 , srv1195550002:84 , srv1206527408:84 , srv1232292528:84 , srv1244535896:84 , srv1249702268:84 , srv1255922287:84 , srv1266031152:84 , srv1290125022:84 , srv130002874:84 , srv1309812616:84 , srv1314197303:84 , srv1333140029:84 , srv1347999558:84 , srv1352952990:84 , srv1420484814:84 , srv1432331255:84 , srv1446330420:84 , srv1446643345:84 , srv1474939720:84 , srv1497715734:84 , srv151249879:84 , srv1526230980:84 , srv1575460046:84 , srv1596337750:84 , srv1620823009:84 , srv1624750065:84 , srv1645755235:84 , srv1692737030:84 , srv1706167536:84 , srv17195000:84 , srv1744871650:84 , srv1746478005:84 , srv1769368093:84 , srv1836143259:84 , srv1840684850:84 , srv193568669:84 , srv1975584991:84 , srv198913401:84 , srv1999815709:84 , srv2015365891:84 , srv2038135109:84 , srv2046673200:84 , srv2063572158:84 , srv2081178293:84 , srv2086405808:84 , srv2130702252:84 , srv227390038:84 , srv252299755:84 , srv281242270:84 , srv288331658:84 , srv29762761:84 , srv312684098:84 , srv334162553:84 , srv345086359:84 , srv34509789:84 , srv35569176:84 , srv37596480:84 , srv376208188:84 , srv394523130:84 , srv451586406:84 , srv498415009:84 , srv499701216:84 , srv535082354:84 , srv543149079:84 , srv556900878:84 , srv577475643:84 , srv594600110:84 , srv601009328:84 , srv602769317:84 , srv620032617:84 , srv621212106:84 , srv636182772:84 , srv64326261:84 , srv676646880:84 , srv715562820:84 , srv718384389:84 , srv73921337:84 , srv800986656:84 , srv810628464:84 , srv859046635:84 , srv8888421:84 , srv910635475:84 , srv91362539:84 , srv921815490:84 , srv941561171:84 , srv947279754:84 , srv95399693:84 , srv960316778:84 , srv994867403:84 , srv591182116:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-04-03 03:53:18,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-03 03:53:18,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1836143259=43, srv947279754=96, srv193568669=45, srv1746478005=41, srv715562820=85, srv198913401=47, srv130002874=18, srv2015365891=49, srv1249702268=14, srv394523130=68, srv1352952990=23, srv1840684850=44, srv73921337=87, srv1420484814=24, srv960316778=98, srv1232292528=12, srv591182116=76, srv620032617=80, srv1018386950=2, srv2130702252=55, srv29762761=60, srv1692737030=37, srv1206527408=11, srv1446643345=27, srv1432331255=25, srv1999815709=48, srv2063572158=52, srv676646880=84, srv1620823009=34, srv1055719027=4, srv312684098=61, srv1309812616=19, srv1975584991=46, srv288331658=59, srv800986656=88, srv1079283912=6, srv1446330420=26, srv718384389=86, srv621212106=81, srv2046673200=51, srv1290125022=17, srv95399693=97, srv1063313261=5, srv1526230980=31, srv1054876834=3, srv2086405808=54, srv1314197303=20, srv1138758271=8, srv1744871650=40, srv376208188=67, srv1004456083=0, srv451586406=69, srv345086359=63, srv34509789=64, srv91362539=93, srv1645755235=36, srv601009328=78, srv556900878=74, srv151249879=30, srv64326261=83, srv1596337750=33, srv2038135109=50, srv498415009=70, srv921815490=94, srv1347999558=22, srv577475643=75, srv1624750065=35, srv8888421=91, srv1497715734=29, srv1474939720=28, srv17195000=39, srv2081178293=53, srv810628464=89, srv602769317=79, srv11391989=9, srv37596480=66, srv1575460046=32, srv1244535896=13, srv1706167536=38, srv1769368093=42, srv281242270=58, srv1083781996=7, srv1195550002=10, srv859046635=90, srv994867403=99, srv543149079=73, srv594600110=77, srv1255922287=15, srv227390038=56, srv334162553=62, srv636182772=82, srv535082354=72, srv941561171=95, srv1266031152=16, srv910635475=92, srv1013659809=1, srv35569176=65, srv499701216=71, srv1333140029=21, srv252299755=57} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-03 03:53:18,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-03 03:53:18,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-03 03:53:18,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-03 03:53:18,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-03 03:53:18,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-03 03:53:18,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-03 03:53:18,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-03 03:53:18,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-03 03:53:18,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-03 03:53:18,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-03 03:53:18,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-03 03:53:18,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-03 03:53:18,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-03 03:53:18,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-03 03:53:18,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-03 03:53:18,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-03 03:53:18,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-03 03:53:18,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-03 03:53:18,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-03 03:53:18,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-03 03:53:18,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-03 03:53:18,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-03 03:53:18,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-03 03:53:18,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-03 03:53:18,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-03 03:53:18,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-03 03:53:18,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-03 03:53:18,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-03 03:53:18,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-03 03:53:18,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-03 03:53:18,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-04-03 03:53:18,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-04-03 03:53:18,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-04-03 03:53:18,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-04-03 03:53:18,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-04-03 03:53:18,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-04-03 03:53:18,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-04-03 03:53:18,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-04-03 03:53:18,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-04-03 03:53:18,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-04-03 03:53:18,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-04-03 03:53:18,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-04-03 03:53:18,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-04-03 03:53:18,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-04-03 03:53:18,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-04-03 03:53:18,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-04-03 03:53:18,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-04-03 03:53:18,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-04-03 03:53:18,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-04-03 03:53:18,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-04-03 03:53:18,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-04-03 03:53:18,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-04-03 03:53:18,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-04-03 03:53:18,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-04-03 03:53:18,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-04-03 03:53:18,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-04-03 03:53:18,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-04-03 03:53:18,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-04-03 03:53:18,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-04-03 03:53:18,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-04-03 03:53:18,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-04-03 03:53:18,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-04-03 03:53:18,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-04-03 03:53:18,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-04-03 03:53:18,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-04-03 03:53:18,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-04-03 03:53:18,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-04-03 03:53:18,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-04-03 03:53:18,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-04-03 03:53:18,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-04-03 03:53:18,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-04-03 03:53:18,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-04-03 03:53:18,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-04-03 03:53:18,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-04-03 03:53:18,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-04-03 03:53:18,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-04-03 03:53:18,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-04-03 03:53:18,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-04-03 03:53:18,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-04-03 03:53:18,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-04-03 03:53:18,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-04-03 03:53:18,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-04-03 03:53:18,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-04-03 03:53:18,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-04-03 03:53:18,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-04-03 03:53:18,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-04-03 03:53:18,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-04-03 03:53:18,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-04-03 03:53:18,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-04-03 03:53:18,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-04-03 03:53:18,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-04-03 03:53:18,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-04-03 03:53:18,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-04-03 03:53:18,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-04-03 03:53:18,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-04-03 03:53:18,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-04-03 03:53:18,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-04-03 03:53:18,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-04-03 03:53:18,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-04-03 03:53:18,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-04-03 03:53:18,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-04-03 03:53:18,964 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 03:53:18,964 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-04-03 03:53:18,965 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-04-03 03:53:19,341 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 753 ms to try 5000 different iterations.  Found a solution that moves 4692 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1752893448875152. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04160365605966457, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054414545490546556, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5213333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04160365605966457, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17083333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23341666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 03:53:19,364 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1769368093:82 , srv1249702268:83 , srv130002874:83 , srv1432331255:83 , srv1596337750:83 , srv2081178293:83 , srv29762761:83 , srv602769317:83 , srv921815490:83 , srv960316778:83 , srv1266031152:84 , srv1309812616:84 , srv1446330420:84 , srv1446643345:84 , srv1526230980:84 , srv2063572158:84 , srv281242270:84 , srv345086359:84 , srv498415009:84 , srv499701216:84 , srv577475643:84 , srv601009328:84 , srv910635475:84 , srv91362539:84 , srv941561171:84 , srv1018386950:85 , srv1255922287:85 , srv1314197303:85 , srv1333140029:85 , srv1352952990:85 , srv1746478005:85 , srv1836143259:85 , srv1975584991:85 , srv2015365891:85 , srv37596480:85 , srv543149079:85 , srv621212106:85 , srv64326261:85 , srv715562820:85 , srv8888421:85 , srv994867403:85 , srv1013659809:86 , srv1206527408:86 , srv1474939720:86 , srv1497715734:86 , srv1620823009:86 , srv1692737030:86 , srv1706167536:86 , srv1744871650:86 , srv193568669:86 , srv198913401:86 , srv288331658:86 , srv556900878:86 , srv676646880:86 , srv73921337:86 , srv95399693:86 , srv1054876834:87 , srv1063313261:87 , srv1079283912:87 , srv1083781996:87 , srv1290125022:87 , srv1575460046:87 , srv1624750065:87 , srv2046673200:87 , srv2086405808:87 , srv2130702252:87 , srv334162553:87 , srv394523130:87 , srv594600110:87 , srv800986656:87 , srv810628464:87 , srv947279754:87 , srv1055719027:88 , srv11391989:88 , srv1420484814:88 , srv151249879:88 , srv1840684850:88 , srv2038135109:88 , srv227390038:88 , srv312684098:88 , srv35569176:88 , srv620032617:88 , srv718384389:88 , srv859046635:88 , srv1195550002:89 , srv1244535896:89 , srv1347999558:89 , srv252299755:89 , srv535082354:89 , srv17195000:90 , srv34509789:90 , srv376208188:90 , srv451586406:90 , srv636182772:90 , srv1138758271:91 , srv1645755235:91 , srv1999815709:91 , srv1232292528:92 , srv1004456083:93 , srv591182116:460 }
2024-04-03 03:53:19,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-03 03:53:19,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1836143259=43, srv947279754=96, srv193568669=45, srv1746478005=41, srv715562820=85, srv198913401=47, srv130002874=18, srv2015365891=49, srv1249702268=14, srv394523130=68, srv1352952990=23, srv1840684850=44, srv73921337=87, srv1420484814=24, srv960316778=98, srv1232292528=12, srv591182116=76, srv620032617=80, srv1018386950=2, srv2130702252=55, srv29762761=60, srv1692737030=37, srv1206527408=11, srv1446643345=27, srv1432331255=25, srv1999815709=48, srv2063572158=52, srv676646880=84, srv1620823009=34, srv1055719027=4, srv312684098=61, srv1309812616=19, srv1975584991=46, srv288331658=59, srv800986656=88, srv1079283912=6, srv1446330420=26, srv718384389=86, srv621212106=81, srv2046673200=51, srv1290125022=17, srv95399693=97, srv1063313261=5, srv1526230980=31, srv1054876834=3, srv2086405808=54, srv1314197303=20, srv1138758271=8, srv1744871650=40, srv376208188=67, srv1004456083=0, srv451586406=69, srv345086359=63, srv34509789=64, srv91362539=93, srv1645755235=36, srv601009328=78, srv556900878=74, srv151249879=30, srv64326261=83, srv1596337750=33, srv2038135109=50, srv498415009=70, srv921815490=94, srv1347999558=22, srv577475643=75, srv1624750065=35, srv8888421=91, srv1497715734=29, srv1474939720=28, srv17195000=39, srv2081178293=53, srv810628464=89, srv602769317=79, srv11391989=9, srv37596480=66, srv1575460046=32, srv1244535896=13, srv1706167536=38, srv1769368093=42, srv281242270=58, srv1083781996=7, srv1195550002=10, srv859046635=90, srv994867403=99, srv543149079=73, srv594600110=77, srv1255922287=15, srv227390038=56, srv334162553=62, srv636182772=82, srv535082354=72, srv941561171=95, srv1266031152=16, srv910635475=92, srv1013659809=1, srv35569176=65, srv499701216=71, srv1333140029=21, srv252299755=57} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-03 03:53:19,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-03 03:53:19,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-03 03:53:19,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-03 03:53:19,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-03 03:53:19,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-03 03:53:19,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-03 03:53:19,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-03 03:53:19,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-03 03:53:19,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-03 03:53:19,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-03 03:53:19,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-03 03:53:19,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-03 03:53:19,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-03 03:53:19,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-03 03:53:19,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-03 03:53:19,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-03 03:53:19,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-03 03:53:19,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-03 03:53:19,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-03 03:53:19,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-03 03:53:19,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-03 03:53:19,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-03 03:53:19,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-03 03:53:19,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-03 03:53:19,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-03 03:53:19,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-03 03:53:19,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-03 03:53:19,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-03 03:53:19,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-03 03:53:19,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-03 03:53:19,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-04-03 03:53:19,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-04-03 03:53:19,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-04-03 03:53:19,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-04-03 03:53:19,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-04-03 03:53:19,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-04-03 03:53:19,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-04-03 03:53:19,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-04-03 03:53:19,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-04-03 03:53:19,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-04-03 03:53:19,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-04-03 03:53:19,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-04-03 03:53:19,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-04-03 03:53:19,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-04-03 03:53:19,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-04-03 03:53:19,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-04-03 03:53:19,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-04-03 03:53:19,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-04-03 03:53:19,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-04-03 03:53:19,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-04-03 03:53:19,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-04-03 03:53:19,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-04-03 03:53:19,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-04-03 03:53:19,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-04-03 03:53:19,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-04-03 03:53:19,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-04-03 03:53:19,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-04-03 03:53:19,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-04-03 03:53:19,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-04-03 03:53:19,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-04-03 03:53:19,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-04-03 03:53:19,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-04-03 03:53:19,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-04-03 03:53:19,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-04-03 03:53:19,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-04-03 03:53:19,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-04-03 03:53:19,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-04-03 03:53:19,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-04-03 03:53:19,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-04-03 03:53:19,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-04-03 03:53:19,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-04-03 03:53:19,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-04-03 03:53:19,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-04-03 03:53:19,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-04-03 03:53:19,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-04-03 03:53:19,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-04-03 03:53:19,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-04-03 03:53:19,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-04-03 03:53:19,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-04-03 03:53:19,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-04-03 03:53:19,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-04-03 03:53:19,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-04-03 03:53:19,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-04-03 03:53:19,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-04-03 03:53:19,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-04-03 03:53:19,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-04-03 03:53:19,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-04-03 03:53:19,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-04-03 03:53:19,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-04-03 03:53:19,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-04-03 03:53:19,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-04-03 03:53:19,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-04-03 03:53:19,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-04-03 03:53:19,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-04-03 03:53:19,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-04-03 03:53:19,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-04-03 03:53:19,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-04-03 03:53:19,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-04-03 03:53:19,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-04-03 03:53:19,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-04-03 03:53:19,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-04-03 03:53:19,582 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04160365605966457, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054414545490546556, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04160365605966457, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17083333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23341666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 03:53:19,583 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-04-03 03:53:19,583 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17525649080131075, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04160365605966457, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054414545490546556, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04160365605966457, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17083333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23341666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-04-03 03:53:19,878 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 439 ms to try 5000 different iterations.  Found a solution that moves 2634 regions; Going from a computed imbalance of 0.17525649080131075 to a new imbalance of 0.014141883498834378. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023738925851302488); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03551133613915883, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2926666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023738925851302488); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0455, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 03:53:19,892 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1004456083:80 , srv1138758271:80 , srv1232292528:80 , srv1645755235:80 , srv1999815709:80 , srv17195000:81 , srv535082354:81 , srv1055719027:82 , srv11391989:82 , srv1244535896:82 , srv34509789:82 , srv35569176:82 , srv1063313261:83 , srv1195550002:83 , srv1290125022:83 , srv1347999558:83 , srv227390038:83 , srv252299755:83 , srv636182772:83 , srv1054876834:84 , srv1079283912:84 , srv1206527408:84 , srv151249879:84 , srv1624750065:84 , srv1840684850:84 , srv198913401:84 , srv2038135109:84 , srv312684098:84 , srv451586406:84 , srv556900878:84 , srv594600110:84 , srv800986656:84 , srv810628464:84 , srv1420484814:85 , srv1474939720:85 , srv1526230980:85 , srv1575460046:85 , srv1692737030:85 , srv1975584991:85 , srv2015365891:85 , srv2130702252:85 , srv334162553:85 , srv376208188:85 , srv64326261:85 , srv676646880:85 , srv715562820:85 , srv718384389:85 , srv95399693:85 , srv1333140029:86 , srv1446643345:86 , srv1620823009:86 , srv1836143259:86 , srv193568669:86 , srv2046673200:86 , srv2086405808:86 , srv543149079:86 , srv1013659809:87 , srv1018386950:87 , srv1083781996:87 , srv1249702268:87 , srv1255922287:87 , srv1266031152:87 , srv1309812616:87 , srv1432331255:87 , srv1497715734:87 , srv1596337750:87 , srv2063572158:87 , srv288331658:87 , srv37596480:87 , srv394523130:87 , srv499701216:87 , srv620032617:87 , srv859046635:87 , srv91362539:87 , srv947279754:87 , srv994867403:87 , srv130002874:88 , srv1314197303:88 , srv1446330420:88 , srv1706167536:88 , srv1744871650:88 , srv281242270:88 , srv345086359:88 , srv498415009:88 , srv577475643:88 , srv8888421:88 , srv910635475:88 , srv921815490:88 , srv1352952990:89 , srv1746478005:89 , srv29762761:89 , srv601009328:89 , srv941561171:89 , srv960316778:89 , srv1769368093:90 , srv602769317:90 , srv73921337:90 , srv2081178293:91 , srv621212106:92 , srv591182116:525 }
2024-04-03 03:53:19,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-03 03:53:19,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1836143259=43, srv947279754=96, srv193568669=45, srv1746478005=41, srv715562820=85, srv198913401=47, srv130002874=18, srv2015365891=49, srv1249702268=14, srv394523130=68, srv1352952990=23, srv1840684850=44, srv73921337=87, srv1420484814=24, srv960316778=98, srv1232292528=12, srv591182116=76, srv620032617=80, srv1018386950=2, srv2130702252=55, srv29762761=60, srv1692737030=37, srv1206527408=11, srv1446643345=27, srv1432331255=25, srv1999815709=48, srv2063572158=52, srv676646880=84, srv1620823009=34, srv1055719027=4, srv312684098=61, srv1309812616=19, srv1975584991=46, srv288331658=59, srv800986656=88, srv1079283912=6, srv1446330420=26, srv718384389=86, srv621212106=81, srv2046673200=51, srv1290125022=17, srv95399693=97, srv1063313261=5, srv1526230980=31, srv1054876834=3, srv2086405808=54, srv1314197303=20, srv1138758271=8, srv1744871650=40, srv376208188=67, srv1004456083=0, srv451586406=69, srv345086359=63, srv34509789=64, srv91362539=93, srv1645755235=36, srv601009328=78, srv556900878=74, srv151249879=30, srv64326261=83, srv1596337750=33, srv2038135109=50, srv498415009=70, srv921815490=94, srv1347999558=22, srv577475643=75, srv1624750065=35, srv8888421=91, srv1497715734=29, srv1474939720=28, srv17195000=39, srv2081178293=53, srv810628464=89, srv602769317=79, srv11391989=9, srv37596480=66, srv1575460046=32, srv1244535896=13, srv1706167536=38, srv1769368093=42, srv281242270=58, srv1083781996=7, srv1195550002=10, srv859046635=90, srv994867403=99, srv543149079=73, srv594600110=77, srv1255922287=15, srv227390038=56, srv334162553=62, srv636182772=82, srv535082354=72, srv941561171=95, srv1266031152=16, srv910635475=92, srv1013659809=1, srv35569176=65, srv499701216=71, srv1333140029=21, srv252299755=57} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-03 03:53:19,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-03 03:53:19,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-03 03:53:19,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-03 03:53:19,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-03 03:53:19,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-03 03:53:19,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-03 03:53:19,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-03 03:53:19,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-03 03:53:19,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-03 03:53:19,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-03 03:53:19,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-03 03:53:19,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-03 03:53:19,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-03 03:53:19,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-03 03:53:19,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-03 03:53:19,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-03 03:53:19,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-03 03:53:19,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-03 03:53:19,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-03 03:53:19,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-03 03:53:19,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-03 03:53:19,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-03 03:53:19,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-03 03:53:19,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-03 03:53:19,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-03 03:53:19,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-03 03:53:19,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-03 03:53:19,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-03 03:53:19,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-03 03:53:19,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-03 03:53:19,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-04-03 03:53:19,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-04-03 03:53:19,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-04-03 03:53:19,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-04-03 03:53:19,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-04-03 03:53:19,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-04-03 03:53:19,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-04-03 03:53:19,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-04-03 03:53:19,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-04-03 03:53:19,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-04-03 03:53:19,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-04-03 03:53:19,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-04-03 03:53:19,933 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-04-03 03:53:19,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-04-03 03:53:19,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-04-03 03:53:19,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-04-03 03:53:19,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-04-03 03:53:19,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-04-03 03:53:19,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-04-03 03:53:19,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-04-03 03:53:19,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-04-03 03:53:19,934 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-04-03 03:53:19,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-04-03 03:53:19,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-04-03 03:53:19,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-04-03 03:53:19,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-04-03 03:53:19,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-04-03 03:53:19,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-04-03 03:53:19,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-04-03 03:53:19,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-04-03 03:53:19,935 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-04-03 03:53:19,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-04-03 03:53:19,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-04-03 03:53:19,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-04-03 03:53:19,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-04-03 03:53:19,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-04-03 03:53:19,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-04-03 03:53:19,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-04-03 03:53:19,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-04-03 03:53:19,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-04-03 03:53:19,936 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-04-03 03:53:19,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-04-03 03:53:19,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-04-03 03:53:19,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-04-03 03:53:19,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-04-03 03:53:19,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-04-03 03:53:19,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-04-03 03:53:19,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-04-03 03:53:19,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-04-03 03:53:19,937 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-04-03 03:53:19,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-04-03 03:53:19,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-04-03 03:53:19,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-04-03 03:53:19,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-04-03 03:53:19,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-04-03 03:53:19,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-04-03 03:53:19,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-04-03 03:53:19,938 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-04-03 03:53:19,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-04-03 03:53:19,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-04-03 03:53:19,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-04-03 03:53:19,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-04-03 03:53:19,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-04-03 03:53:19,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-04-03 03:53:19,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-04-03 03:53:19,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-04-03 03:53:19,939 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-04-03 03:53:19,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-04-03 03:53:19,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-04-03 03:53:19,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-04-03 03:53:19,940 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-04-03 03:53:20,092 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023738925851302488); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03551133613915883, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023738925851302488); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0455, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 03:53:20,093 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-04-03 03:53:20,093 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014123439836630081, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023738925851302488); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03551133613915883, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023738925851302488); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0455, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-04-03 03:53:20,391 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 425 ms to try 5000 different iterations.  Found a solution that moves 976 regions; Going from a computed imbalance of 0.014123439836630081 to a new imbalance of 0.0010683098452648715. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009814338340508634); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011976407785072427); MoveCostFunction : (multiplier=7.0, imbalance=0.10844444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.009814338340508634); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 03:53:20,395 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv621212106:81 , srv1352952990:82 , srv451586406:82 , srv620032617:82 , srv73921337:82 , srv1706167536:83 , srv1744871650:83 , srv2081178293:83 , srv376208188:83 , srv718384389:83 , srv859046635:83 , srv941561171:83 , srv947279754:83 , srv1013659809:84 , srv1083781996:84 , srv1195550002:84 , srv1232292528:84 , srv1314197303:84 , srv1347999558:84 , srv1420484814:84 , srv1840684850:84 , srv281242270:84 , srv288331658:84 , srv29762761:84 , srv334162553:84 , srv34509789:84 , srv394523130:84 , srv577475643:84 , srv602769317:84 , srv636182772:84 , srv994867403:84 , srv1018386950:85 , srv1054876834:85 , srv1138758271:85 , srv130002874:85 , srv1497715734:85 , srv1575460046:85 , srv1620823009:85 , srv193568669:85 , srv1999815709:85 , srv2046673200:85 , srv2086405808:85 , srv252299755:85 , srv345086359:85 , srv37596480:85 , srv498415009:85 , srv499701216:85 , srv594600110:85 , srv601009328:85 , srv810628464:85 , srv8888421:85 , srv910635475:85 , srv960316778:85 , srv1004456083:86 , srv11391989:86 , srv1244535896:86 , srv1249702268:86 , srv1255922287:86 , srv1266031152:86 , srv1290125022:86 , srv1309812616:86 , srv1333140029:86 , srv1446330420:86 , srv1474939720:86 , srv1692737030:86 , srv17195000:86 , srv1746478005:86 , srv1769368093:86 , srv1836143259:86 , srv2130702252:86 , srv227390038:86 , srv312684098:86 , srv676646880:86 , srv800986656:86 , srv91362539:86 , srv95399693:86 , srv1055719027:87 , srv1079283912:87 , srv1432331255:87 , srv1446643345:87 , srv151249879:87 , srv1526230980:87 , srv1596337750:87 , srv1645755235:87 , srv1975584991:87 , srv198913401:87 , srv2015365891:87 , srv2038135109:87 , srv2063572158:87 , srv535082354:87 , srv543149079:87 , srv556900878:87 , srv64326261:87 , srv715562820:87 , srv1206527408:88 , srv1624750065:88 , srv35569176:88 , srv921815490:88 , srv1063313261:89 , srv591182116:560 }
2024-04-03 03:53:20,402 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-03 03:53:20,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1836143259=43, srv947279754=96, srv193568669=45, srv1746478005=41, srv715562820=85, srv198913401=47, srv130002874=18, srv2015365891=49, srv1249702268=14, srv394523130=68, srv1352952990=23, srv1840684850=44, srv73921337=87, srv1420484814=24, srv960316778=98, srv1232292528=12, srv591182116=76, srv620032617=80, srv1018386950=2, srv2130702252=55, srv29762761=60, srv1692737030=37, srv1206527408=11, srv1446643345=27, srv1432331255=25, srv1999815709=48, srv2063572158=52, srv676646880=84, srv1620823009=34, srv1055719027=4, srv312684098=61, srv1309812616=19, srv1975584991=46, srv288331658=59, srv800986656=88, srv1079283912=6, srv1446330420=26, srv718384389=86, srv621212106=81, srv2046673200=51, srv1290125022=17, srv95399693=97, srv1063313261=5, srv1526230980=31, srv1054876834=3, srv2086405808=54, srv1314197303=20, srv1138758271=8, srv1744871650=40, srv376208188=67, srv1004456083=0, srv451586406=69, srv345086359=63, srv34509789=64, srv91362539=93, srv1645755235=36, srv601009328=78, srv556900878=74, srv151249879=30, srv64326261=83, srv1596337750=33, srv2038135109=50, srv498415009=70, srv921815490=94, srv1347999558=22, srv577475643=75, srv1624750065=35, srv8888421=91, srv1497715734=29, srv1474939720=28, srv17195000=39, srv2081178293=53, srv810628464=89, srv602769317=79, srv11391989=9, srv37596480=66, srv1575460046=32, srv1244535896=13, srv1706167536=38, srv1769368093=42, srv281242270=58, srv1083781996=7, srv1195550002=10, srv859046635=90, srv994867403=99, srv543149079=73, srv594600110=77, srv1255922287=15, srv227390038=56, srv334162553=62, srv636182772=82, srv535082354=72, srv941561171=95, srv1266031152=16, srv910635475=92, srv1013659809=1, srv35569176=65, srv499701216=71, srv1333140029=21, srv252299755=57} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-03 03:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-03 03:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-03 03:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-03 03:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-03 03:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-03 03:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-03 03:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-03 03:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-03 03:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-03 03:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-03 03:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-03 03:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-03 03:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-03 03:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-03 03:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-03 03:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-03 03:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-03 03:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-03 03:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-03 03:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-03 03:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-03 03:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-03 03:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-03 03:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-03 03:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-03 03:53:20,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-03 03:53:20,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-03 03:53:20,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-03 03:53:20,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-03 03:53:20,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-03 03:53:20,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-04-03 03:53:20,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-04-03 03:53:20,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-04-03 03:53:20,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-04-03 03:53:20,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-04-03 03:53:20,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-04-03 03:53:20,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-04-03 03:53:20,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-04-03 03:53:20,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-04-03 03:53:20,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-04-03 03:53:20,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-04-03 03:53:20,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-04-03 03:53:20,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-04-03 03:53:20,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-04-03 03:53:20,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-04-03 03:53:20,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-04-03 03:53:20,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-04-03 03:53:20,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-04-03 03:53:20,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-04-03 03:53:20,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-04-03 03:53:20,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-04-03 03:53:20,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-04-03 03:53:20,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-04-03 03:53:20,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-04-03 03:53:20,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-04-03 03:53:20,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-04-03 03:53:20,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-04-03 03:53:20,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-04-03 03:53:20,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-04-03 03:53:20,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-04-03 03:53:20,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-04-03 03:53:20,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-04-03 03:53:20,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-04-03 03:53:20,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-04-03 03:53:20,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-04-03 03:53:20,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-04-03 03:53:20,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-04-03 03:53:20,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-04-03 03:53:20,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-04-03 03:53:20,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-04-03 03:53:20,429 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-04-03 03:53:20,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-04-03 03:53:20,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-04-03 03:53:20,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-04-03 03:53:20,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-04-03 03:53:20,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-04-03 03:53:20,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-04-03 03:53:20,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-04-03 03:53:20,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-04-03 03:53:20,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-04-03 03:53:20,430 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-04-03 03:53:20,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-04-03 03:53:20,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-04-03 03:53:20,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-04-03 03:53:20,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-04-03 03:53:20,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-04-03 03:53:20,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-04-03 03:53:20,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-04-03 03:53:20,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-04-03 03:53:20,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-04-03 03:53:20,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-04-03 03:53:20,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-04-03 03:53:20,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-04-03 03:53:20,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-04-03 03:53:20,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-04-03 03:53:20,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-04-03 03:53:20,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-04-03 03:53:20,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-04-03 03:53:20,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-04-03 03:53:20,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-04-03 03:53:20,433 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-04-03 03:53:20,583 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009814338340508634); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011976407785072427); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.009814338340508634); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 03:53:20,584 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-04-03 03:53:20,587 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0010614757471967644, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009814338340508634); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011976407785072427); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.009814338340508634); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.833333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004833333333333334); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-04-03 03:53:21,036 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 590 ms to try 5000 different iterations.  Found a solution that moves 314 regions; Going from a computed imbalance of 0.0010614757471967644 to a new imbalance of 4.548775704762469E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024158102511315956); MoveCostFunction : (multiplier=7.0, imbalance=0.034888888888888886, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 03:53:21,038 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1746478005:82 , srv1004456083:83 , srv151249879:83 , srv1624750065:83 , srv2038135109:83 , srv2063572158:83 , srv921815490:83 , srv1055719027:84 , srv1063313261:84 , srv1079283912:84 , srv1083781996:84 , srv1206527408:84 , srv1255922287:84 , srv1314197303:84 , srv1446330420:84 , srv1497715734:84 , srv1575460046:84 , srv1645755235:84 , srv1769368093:84 , srv2046673200:84 , srv2086405808:84 , srv2130702252:84 , srv312684098:84 , srv35569176:84 , srv376208188:84 , srv394523130:84 , srv535082354:84 , srv543149079:84 , srv601009328:84 , srv621212106:84 , srv73921337:84 , srv8888421:84 , srv1018386950:85 , srv1232292528:85 , srv1244535896:85 , srv1266031152:85 , srv1309812616:85 , srv1420484814:85 , srv1432331255:85 , srv1446643345:85 , srv1474939720:85 , srv1596337750:85 , srv1692737030:85 , srv1706167536:85 , srv17195000:85 , srv1744871650:85 , srv1836143259:85 , srv1840684850:85 , srv193568669:85 , srv1975584991:85 , srv1999815709:85 , srv2081178293:85 , srv227390038:85 , srv252299755:85 , srv288331658:85 , srv29762761:85 , srv345086359:85 , srv37596480:85 , srv498415009:85 , srv556900878:85 , srv594600110:85 , srv602769317:85 , srv636182772:85 , srv64326261:85 , srv676646880:85 , srv715562820:85 , srv800986656:85 , srv859046635:85 , srv910635475:85 , srv91362539:85 , srv947279754:85 , srv960316778:85 , srv1013659809:86 , srv1054876834:86 , srv1138758271:86 , srv11391989:86 , srv1195550002:86 , srv1249702268:86 , srv1290125022:86 , srv1333140029:86 , srv1347999558:86 , srv1352952990:86 , srv1526230980:86 , srv1620823009:86 , srv198913401:86 , srv2015365891:86 , srv281242270:86 , srv334162553:86 , srv34509789:86 , srv451586406:86 , srv499701216:86 , srv577475643:86 , srv620032617:86 , srv718384389:86 , srv810628464:86 , srv941561171:86 , srv95399693:86 , srv994867403:86 , srv130002874:87 , srv591182116:597 }
2024-04-03 03:53:21,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-03 03:53:21,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1836143259=43, srv947279754=96, srv193568669=45, srv1746478005=41, srv715562820=85, srv198913401=47, srv130002874=18, srv2015365891=49, srv1249702268=14, srv394523130=68, srv1352952990=23, srv1840684850=44, srv73921337=87, srv1420484814=24, srv960316778=98, srv1232292528=12, srv591182116=76, srv620032617=80, srv1018386950=2, srv2130702252=55, srv29762761=60, srv1692737030=37, srv1206527408=11, srv1446643345=27, srv1432331255=25, srv1999815709=48, srv2063572158=52, srv676646880=84, srv1620823009=34, srv1055719027=4, srv312684098=61, srv1309812616=19, srv1975584991=46, srv288331658=59, srv800986656=88, srv1079283912=6, srv1446330420=26, srv718384389=86, srv621212106=81, srv2046673200=51, srv1290125022=17, srv95399693=97, srv1063313261=5, srv1526230980=31, srv1054876834=3, srv2086405808=54, srv1314197303=20, srv1138758271=8, srv1744871650=40, srv376208188=67, srv1004456083=0, srv451586406=69, srv345086359=63, srv34509789=64, srv91362539=93, srv1645755235=36, srv601009328=78, srv556900878=74, srv151249879=30, srv64326261=83, srv1596337750=33, srv2038135109=50, srv498415009=70, srv921815490=94, srv1347999558=22, srv577475643=75, srv1624750065=35, srv8888421=91, srv1497715734=29, srv1474939720=28, srv17195000=39, srv2081178293=53, srv810628464=89, srv602769317=79, srv11391989=9, srv37596480=66, srv1575460046=32, srv1244535896=13, srv1706167536=38, srv1769368093=42, srv281242270=58, srv1083781996=7, srv1195550002=10, srv859046635=90, srv994867403=99, srv543149079=73, srv594600110=77, srv1255922287=15, srv227390038=56, srv334162553=62, srv636182772=82, srv535082354=72, srv941561171=95, srv1266031152=16, srv910635475=92, srv1013659809=1, srv35569176=65, srv499701216=71, srv1333140029=21, srv252299755=57} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-03 03:53:21,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-03 03:53:21,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-03 03:53:21,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-03 03:53:21,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-03 03:53:21,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-03 03:53:21,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-03 03:53:21,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-03 03:53:21,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-03 03:53:21,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-03 03:53:21,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-03 03:53:21,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-03 03:53:21,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-03 03:53:21,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-03 03:53:21,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-03 03:53:21,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-03 03:53:21,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-03 03:53:21,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-03 03:53:21,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-03 03:53:21,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-03 03:53:21,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-03 03:53:21,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-03 03:53:21,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-03 03:53:21,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-03 03:53:21,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-03 03:53:21,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-03 03:53:21,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-03 03:53:21,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-03 03:53:21,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-03 03:53:21,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-03 03:53:21,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-03 03:53:21,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-04-03 03:53:21,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-04-03 03:53:21,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-04-03 03:53:21,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-04-03 03:53:21,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-04-03 03:53:21,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-04-03 03:53:21,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-04-03 03:53:21,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-04-03 03:53:21,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-04-03 03:53:21,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-04-03 03:53:21,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-04-03 03:53:21,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-04-03 03:53:21,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-04-03 03:53:21,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-04-03 03:53:21,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-04-03 03:53:21,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-04-03 03:53:21,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-04-03 03:53:21,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-04-03 03:53:21,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-04-03 03:53:21,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-04-03 03:53:21,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-04-03 03:53:21,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-04-03 03:53:21,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-04-03 03:53:21,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-04-03 03:53:21,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-04-03 03:53:21,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-04-03 03:53:21,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-04-03 03:53:21,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-04-03 03:53:21,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-04-03 03:53:21,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-04-03 03:53:21,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-04-03 03:53:21,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-04-03 03:53:21,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-04-03 03:53:21,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-04-03 03:53:21,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-04-03 03:53:21,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-04-03 03:53:21,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-04-03 03:53:21,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-04-03 03:53:21,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-04-03 03:53:21,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-04-03 03:53:21,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-04-03 03:53:21,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-04-03 03:53:21,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-04-03 03:53:21,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-04-03 03:53:21,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-04-03 03:53:21,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-04-03 03:53:21,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-04-03 03:53:21,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-04-03 03:53:21,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-04-03 03:53:21,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-04-03 03:53:21,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-04-03 03:53:21,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-04-03 03:53:21,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-04-03 03:53:21,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-04-03 03:53:21,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-04-03 03:53:21,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-04-03 03:53:21,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-04-03 03:53:21,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-04-03 03:53:21,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-04-03 03:53:21,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-04-03 03:53:21,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-04-03 03:53:21,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-04-03 03:53:21,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-04-03 03:53:21,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-04-03 03:53:21,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-04-03 03:53:21,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-04-03 03:53:21,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-04-03 03:53:21,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-04-03 03:53:21,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-04-03 03:53:21,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-04-03 03:53:21,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-04-03 03:53:21,185 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024158102511315956); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 03:53:21,185 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-04-03 03:53:21,186 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.328908205440178E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0024158102511315956); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-04-03 03:53:21,419 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 336 ms to try 5000 different iterations.  Found a solution that moves 43 regions; Going from a computed imbalance of 4.328908205440178E-5 to a new imbalance of 7.68884993569022E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0047777777777777775); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 03:53:21,420 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1013659809:83 , srv1063313261:83 , srv130002874:83 , srv1333140029:83 , srv1620823009:83 , srv198913401:83 , srv2015365891:83 , srv620032617:83 , srv859046635:83 , srv95399693:83 , srv1004456083:84 , srv1018386950:84 , srv1054876834:84 , srv1079283912:84 , srv1083781996:84 , srv1138758271:84 , srv11391989:84 , srv1195550002:84 , srv1206527408:84 , srv1244535896:84 , srv1249702268:84 , srv1255922287:84 , srv1266031152:84 , srv1290125022:84 , srv1309812616:84 , srv1347999558:84 , srv1352952990:84 , srv1420484814:84 , srv1432331255:84 , srv1446330420:84 , srv1446643345:84 , srv1474939720:84 , srv1497715734:84 , srv151249879:84 , srv1526230980:84 , srv1596337750:84 , srv1624750065:84 , srv1645755235:84 , srv1692737030:84 , srv1706167536:84 , srv17195000:84 , srv1744871650:84 , srv1746478005:84 , srv1769368093:84 , srv1836143259:84 , srv193568669:84 , srv1975584991:84 , srv2038135109:84 , srv2046673200:84 , srv2081178293:84 , srv2086405808:84 , srv2130702252:84 , srv227390038:84 , srv252299755:84 , srv281242270:84 , srv288331658:84 , srv312684098:84 , srv334162553:84 , srv345086359:84 , srv34509789:84 , srv35569176:84 , srv37596480:84 , srv376208188:84 , srv394523130:84 , srv451586406:84 , srv498415009:84 , srv499701216:84 , srv543149079:84 , srv556900878:84 , srv577475643:84 , srv601009328:84 , srv602769317:84 , srv621212106:84 , srv636182772:84 , srv64326261:84 , srv676646880:84 , srv715562820:84 , srv718384389:84 , srv73921337:84 , srv800986656:84 , srv810628464:84 , srv8888421:84 , srv910635475:84 , srv91362539:84 , srv921815490:84 , srv941561171:84 , srv947279754:84 , srv960316778:84 , srv994867403:84 , srv1055719027:85 , srv1232292528:85 , srv1314197303:85 , srv1575460046:85 , srv1840684850:85 , srv1999815709:85 , srv2063572158:85 , srv29762761:85 , srv535082354:85 , srv594600110:85 , srv591182116:684 }
2024-04-03 03:53:21,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-03 03:53:21,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1836143259=43, srv947279754=96, srv193568669=45, srv1746478005=41, srv715562820=85, srv198913401=47, srv130002874=18, srv2015365891=49, srv1249702268=14, srv394523130=68, srv1352952990=23, srv1840684850=44, srv73921337=87, srv1420484814=24, srv960316778=98, srv1232292528=12, srv591182116=76, srv620032617=80, srv1018386950=2, srv2130702252=55, srv29762761=60, srv1692737030=37, srv1206527408=11, srv1446643345=27, srv1432331255=25, srv1999815709=48, srv2063572158=52, srv676646880=84, srv1620823009=34, srv1055719027=4, srv312684098=61, srv1309812616=19, srv1975584991=46, srv288331658=59, srv800986656=88, srv1079283912=6, srv1446330420=26, srv718384389=86, srv621212106=81, srv2046673200=51, srv1290125022=17, srv95399693=97, srv1063313261=5, srv1526230980=31, srv1054876834=3, srv2086405808=54, srv1314197303=20, srv1138758271=8, srv1744871650=40, srv376208188=67, srv1004456083=0, srv451586406=69, srv345086359=63, srv34509789=64, srv91362539=93, srv1645755235=36, srv601009328=78, srv556900878=74, srv151249879=30, srv64326261=83, srv1596337750=33, srv2038135109=50, srv498415009=70, srv921815490=94, srv1347999558=22, srv577475643=75, srv1624750065=35, srv8888421=91, srv1497715734=29, srv1474939720=28, srv17195000=39, srv2081178293=53, srv810628464=89, srv602769317=79, srv11391989=9, srv37596480=66, srv1575460046=32, srv1244535896=13, srv1706167536=38, srv1769368093=42, srv281242270=58, srv1083781996=7, srv1195550002=10, srv859046635=90, srv994867403=99, srv543149079=73, srv594600110=77, srv1255922287=15, srv227390038=56, srv334162553=62, srv636182772=82, srv535082354=72, srv941561171=95, srv1266031152=16, srv910635475=92, srv1013659809=1, srv35569176=65, srv499701216=71, srv1333140029=21, srv252299755=57} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-03 03:53:21,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-03 03:53:21,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-03 03:53:21,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-03 03:53:21,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-03 03:53:21,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-03 03:53:21,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-03 03:53:21,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-03 03:53:21,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-03 03:53:21,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-03 03:53:21,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-03 03:53:21,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-03 03:53:21,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-03 03:53:21,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-03 03:53:21,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-03 03:53:21,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-03 03:53:21,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-03 03:53:21,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-03 03:53:21,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-03 03:53:21,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-03 03:53:21,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-03 03:53:21,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-03 03:53:21,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-03 03:53:21,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-03 03:53:21,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-03 03:53:21,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-03 03:53:21,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-03 03:53:21,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-03 03:53:21,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-03 03:53:21,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-03 03:53:21,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-03 03:53:21,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-04-03 03:53:21,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-04-03 03:53:21,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-04-03 03:53:21,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-04-03 03:53:21,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-04-03 03:53:21,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-04-03 03:53:21,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-04-03 03:53:21,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-04-03 03:53:21,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-04-03 03:53:21,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-04-03 03:53:21,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-04-03 03:53:21,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-04-03 03:53:21,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-04-03 03:53:21,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-04-03 03:53:21,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-04-03 03:53:21,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-04-03 03:53:21,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-04-03 03:53:21,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-04-03 03:53:21,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-04-03 03:53:21,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-04-03 03:53:21,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-04-03 03:53:21,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-04-03 03:53:21,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-04-03 03:53:21,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-04-03 03:53:21,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-04-03 03:53:21,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-04-03 03:53:21,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-04-03 03:53:21,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-04-03 03:53:21,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-04-03 03:53:21,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-04-03 03:53:21,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-04-03 03:53:21,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-04-03 03:53:21,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-04-03 03:53:21,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-04-03 03:53:21,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-04-03 03:53:21,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-04-03 03:53:21,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-04-03 03:53:21,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-04-03 03:53:21,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-04-03 03:53:21,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-04-03 03:53:21,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-04-03 03:53:21,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-04-03 03:53:21,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-04-03 03:53:21,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-04-03 03:53:21,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-04-03 03:53:21,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-04-03 03:53:21,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-04-03 03:53:21,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-04-03 03:53:21,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-04-03 03:53:21,545 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 03:53:21,545 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1013659809:83 , srv1063313261:83 , srv130002874:83 , srv1333140029:83 , srv1620823009:83 , srv198913401:83 , srv2015365891:83 , srv620032617:83 , srv859046635:83 , srv95399693:83 , srv1004456083:84 , srv1018386950:84 , srv1054876834:84 , srv1079283912:84 , srv1083781996:84 , srv1138758271:84 , srv11391989:84 , srv1195550002:84 , srv1206527408:84 , srv1244535896:84 , srv1249702268:84 , srv1255922287:84 , srv1266031152:84 , srv1290125022:84 , srv1309812616:84 , srv1347999558:84 , srv1352952990:84 , srv1420484814:84 , srv1432331255:84 , srv1446330420:84 , srv1446643345:84 , srv1474939720:84 , srv1497715734:84 , srv151249879:84 , srv1526230980:84 , srv1596337750:84 , srv1624750065:84 , srv1645755235:84 , srv1692737030:84 , srv1706167536:84 , srv17195000:84 , srv1744871650:84 , srv1746478005:84 , srv1769368093:84 , srv1836143259:84 , srv193568669:84 , srv1975584991:84 , srv2038135109:84 , srv2046673200:84 , srv2081178293:84 , srv2086405808:84 , srv2130702252:84 , srv227390038:84 , srv252299755:84 , srv281242270:84 , srv288331658:84 , srv312684098:84 , srv334162553:84 , srv345086359:84 , srv34509789:84 , srv35569176:84 , srv37596480:84 , srv376208188:84 , srv394523130:84 , srv451586406:84 , srv498415009:84 , srv499701216:84 , srv543149079:84 , srv556900878:84 , srv577475643:84 , srv601009328:84 , srv602769317:84 , srv621212106:84 , srv636182772:84 , srv64326261:84 , srv676646880:84 , srv715562820:84 , srv718384389:84 , srv73921337:84 , srv800986656:84 , srv810628464:84 , srv8888421:84 , srv910635475:84 , srv91362539:84 , srv921815490:84 , srv941561171:84 , srv947279754:84 , srv960316778:84 , srv994867403:84 , srv1055719027:85 , srv1232292528:85 , srv1314197303:85 , srv1575460046:85 , srv1840684850:85 , srv1999815709:85 , srv2063572158:85 , srv29762761:85 , srv535082354:85 , srv594600110:85 , srv591182116:684 }
2024-04-03 03:53:21,571 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=316 (was 300) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=10506 (was 11251)
2024-04-03 03:53:21,580 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=316, ProcessCount=170, AvailableMemoryMB=10506
2024-04-03 03:53:21,581 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-04-03 03:53:21,581 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-04-03 03:53:21,596 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1062282932:84 , srv1101058732:84 , srv126767695:84 , srv1362670888:84 , srv1456470222:84 , srv1546523516:84 , srv1618657163:84 , srv1625720814:84 , srv165786671:84 , srv1689923981:84 , srv1698506928:84 , srv181194694:84 , srv1923653316:84 , srv2004402994:84 , srv20298492:84 , srv23726253:84 , srv288664949:84 , srv425828179:84 , srv44258572:84 , srv549026472:84 , srv607747637:84 , srv654671310:84 , srv660199621:84 , srv670667862:84 , srv700647805:84 , srv701099866:84 , srv702387839:84 , srv707009589:84 , srv758355185:84 , srv721644665:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-04-03 03:53:21,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-03 03:53:21,597 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv425828179=17, srv1062282932=0, srv44258572=18, srv758355185=29, srv721644665=28, srv1923653316=12, srv707009589=27, srv1698506928=10, srv549026472=19, srv1362670888=3, srv1625720814=7, srv1689923981=9, srv2004402994=13, srv654671310=21, srv288664949=16, srv702387839=26, srv1618657163=6, srv181194694=11, srv1456470222=4, srv165786671=8, srv20298492=14, srv23726253=15, srv670667862=23, srv126767695=2, srv607747637=20, srv660199621=22, srv701099866=25, srv1546523516=5, srv700647805=24, srv1101058732=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-03 03:53:21,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-03 03:53:21,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-03 03:53:21,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-03 03:53:21,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-03 03:53:21,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-03 03:53:21,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-03 03:53:21,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-03 03:53:21,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-03 03:53:21,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-03 03:53:21,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-03 03:53:21,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-03 03:53:21,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-03 03:53:21,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-03 03:53:21,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-03 03:53:21,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-03 03:53:21,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-03 03:53:21,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-03 03:53:21,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-03 03:53:21,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-03 03:53:21,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-03 03:53:21,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-03 03:53:21,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-03 03:53:21,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-03 03:53:21,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-03 03:53:21,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-03 03:53:21,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-03 03:53:21,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-03 03:53:21,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-03 03:53:21,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-03 03:53:21,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-03 03:53:21,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-04-03 03:53:21,624 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1021317721794313, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 03:53:21,624 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912514857143508, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1021317721794313, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-04-03 03:55:21,611 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5845816 different iterations.  Found a solution that moves 1853 regions; Going from a computed imbalance of 0.9912514857143508 to a new imbalance of 4.324994439959734E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6862962962962963, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 03:55:21,617 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1062282932:90 , srv1101058732:90 , srv126767695:90 , srv1362670888:90 , srv1456470222:90 , srv1546523516:90 , srv1618657163:90 , srv1625720814:90 , srv165786671:90 , srv1689923981:90 , srv1698506928:90 , srv181194694:90 , srv1923653316:90 , srv2004402994:90 , srv20298492:90 , srv23726253:90 , srv288664949:90 , srv425828179:90 , srv44258572:90 , srv549026472:90 , srv607747637:90 , srv654671310:90 , srv660199621:90 , srv670667862:90 , srv700647805:90 , srv701099866:90 , srv702387839:90 , srv707009589:90 , srv721644665:90 , srv758355185:90 }
2024-04-03 03:55:21,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-03 03:55:21,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv425828179=17, srv1062282932=0, srv44258572=18, srv758355185=29, srv721644665=28, srv1923653316=12, srv707009589=27, srv1698506928=10, srv549026472=19, srv1362670888=3, srv1625720814=7, srv1689923981=9, srv2004402994=13, srv654671310=21, srv288664949=16, srv702387839=26, srv1618657163=6, srv181194694=11, srv1456470222=4, srv165786671=8, srv20298492=14, srv23726253=15, srv670667862=23, srv126767695=2, srv607747637=20, srv660199621=22, srv701099866=25, srv1546523516=5, srv700647805=24, srv1101058732=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-03 03:55:21,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-03 03:55:21,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-03 03:55:21,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-03 03:55:21,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-03 03:55:21,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-03 03:55:21,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-03 03:55:21,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-03 03:55:21,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-03 03:55:21,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-03 03:55:21,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-03 03:55:21,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-03 03:55:21,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-03 03:55:21,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-03 03:55:21,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-03 03:55:21,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-03 03:55:21,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-03 03:55:21,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-03 03:55:21,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-03 03:55:21,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-03 03:55:21,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-03 03:55:21,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-03 03:55:21,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-03 03:55:21,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-03 03:55:21,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-03 03:55:21,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-03 03:55:21,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-03 03:55:21,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-03 03:55:21,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-03 03:55:21,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-03 03:55:21,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-03 03:55:21,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-04-03 03:55:21,635 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-03 03:55:21,635 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1062282932:90 , srv1101058732:90 , srv126767695:90 , srv1362670888:90 , srv1456470222:90 , srv1546523516:90 , srv1618657163:90 , srv1625720814:90 , srv165786671:90 , srv1689923981:90 , srv1698506928:90 , srv181194694:90 , srv1923653316:90 , srv2004402994:90 , srv20298492:90 , srv23726253:90 , srv288664949:90 , srv425828179:90 , srv44258572:90 , srv549026472:90 , srv607747637:90 , srv654671310:90 , srv660199621:90 , srv670667862:90 , srv700647805:90 , srv701099866:90 , srv702387839:90 , srv707009589:90 , srv721644665:90 , srv758355185:90 }
2024-04-03 03:55:21,649 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=264 (was 316), ProcessCount=169 (was 170), AvailableMemoryMB=9766 (was 10506)