2024-03-08 15:53:15,956 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-03-08 15:53:16,289 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-03-08 15:53:16,409 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-03-08 15:53:16,478 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-08 15:53:16,486 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-08 15:53:16,501 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=250, ProcessCount=170, AvailableMemoryMB=9926
2024-03-08 15:53:16,503 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-08 15:53:16,503 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-08 15:53:16,634 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1014595466:84 , srv1032840094:84 , srv1078668790:84 , srv1108208098:84 , srv1124822332:84 , srv1127338266:84 , srv1154814852:84 , srv1186564537:84 , srv1228861660:84 , srv1249697444:84 , srv128297666:84 , srv1306526914:84 , srv1355508237:84 , srv1373073360:84 , srv1374035793:84 , srv1381211730:84 , srv1383364875:84 , srv1391623499:84 , srv1400527951:84 , srv1412546130:84 , srv1426941501:84 , srv1427938932:84 , srv1446740569:84 , srv1465039067:84 , srv1491685040:84 , srv1492116257:84 , srv1502649377:84 , srv1565757425:84 , srv1574497805:84 , srv157832699:84 , srv1650519673:84 , srv1697228349:84 , srv1700565924:84 , srv1713337116:84 , srv1730208776:84 , srv1737966533:84 , srv1741420780:84 , srv1753471837:84 , srv1756127224:84 , srv1761851683:84 , srv1804012614:84 , srv1809566825:84 , srv1818126107:84 , srv1843337773:84 , srv1865606187:84 , srv1893396910:84 , srv1949605675:84 , srv2011476985:84 , srv2016469725:84 , srv2016495454:84 , srv20254933:84 , srv2030182528:84 , srv2034727245:84 , srv2119853453:84 , srv2142630400:84 , srv219280996:84 , srv220167258:84 , srv229363422:84 , srv232931826:84 , srv277752129:84 , srv298559727:84 , srv322188390:84 , srv326780280:84 , srv328920534:84 , srv334859030:84 , srv387586074:84 , srv407735636:84 , srv454757238:84 , srv455253557:84 , srv510543993:84 , srv548611336:84 , srv550741197:84 , srv572408198:84 , srv574858268:84 , srv618881394:84 , srv648162819:84 , srv65630548:84 , srv666071834:84 , srv674603265:84 , srv699556681:84 , srv717987243:84 , srv739939434:84 , srv759026509:84 , srv805778281:84 , srv818353502:84 , srv818671758:84 , srv836332868:84 , srv851267417:84 , srv895249072:84 , srv896680763:84 , srv903623104:84 , srv909470651:84 , srv914662246:84 , srv923668394:84 , srv925201202:84 , srv947988749:84 , srv948806881:84 , srv950019898:84 , srv978625798:84 , srv449688332:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-03-08 15:53:16,643 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-08 15:53:16,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1306526914=11, srv455253557=69, srv2011476985=47, srv277752129=59, srv1383364875=16, srv1154814852=6, srv1032840094=1, srv2119853453=53, srv759026509=83, srv232931826=58, srv1465039067=23, srv2142630400=54, srv925201202=95, srv1809566825=41, srv1391623499=17, srv666071834=78, srv1893396910=45, srv1374035793=14, srv1446740569=22, srv1753471837=37, srv2016469725=48, srv328920534=63, srv896680763=90, srv978625798=99, srv157832699=29, srv1565757425=27, srv895249072=89, srv1426941501=20, srv1186564537=7, srv950019898=98, srv510543993=70, srv914662246=93, srv1843337773=43, srv1249697444=9, srv851267417=88, srv1741420780=36, srv548611336=71, srv699556681=80, srv1804012614=40, srv407735636=66, srv903623104=91, srv1491685040=24, srv229363422=57, srv550741197=72, srv1697228349=31, srv1730208776=34, srv20254933=50, srv2016495454=49, srv387586074=65, srv1700565924=32, srv1400527951=18, srv1737966533=35, srv572408198=73, srv1761851683=39, srv923668394=94, srv1355508237=12, srv454757238=68, srv1228861660=8, srv648162819=76, srv909470651=92, srv1108208098=3, srv449688332=67, srv326780280=62, srv1949605675=46, srv1373073360=13, srv220167258=56, srv574858268=74, srv1574497805=28, srv1127338266=5, srv674603265=79, srv739939434=82, srv818671758=86, srv947988749=96, srv1014595466=0, srv298559727=60, srv322188390=61, srv1078668790=2, srv334859030=64, srv818353502=85, srv1818126107=42, srv1124822332=4, srv717987243=81, srv65630548=77, srv1381211730=15, srv1713337116=33, srv219280996=55, srv2030182528=51, srv618881394=75, srv1865606187=44, srv1492116257=25, srv1650519673=30, srv1427938932=21, srv805778281=84, srv2034727245=52, srv1756127224=38, srv128297666=10, srv948806881=97, srv1412546130=19, srv1502649377=26, srv836332868=87} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-08 15:53:16,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-08 15:53:16,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-08 15:53:16,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-08 15:53:16,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-08 15:53:16,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-08 15:53:16,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-08 15:53:16,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-08 15:53:16,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-08 15:53:16,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-08 15:53:16,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-08 15:53:16,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-08 15:53:16,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-08 15:53:16,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-08 15:53:16,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-08 15:53:16,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-08 15:53:16,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-08 15:53:16,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-08 15:53:16,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-08 15:53:16,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-08 15:53:16,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-08 15:53:16,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-08 15:53:16,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-08 15:53:16,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-08 15:53:16,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-08 15:53:16,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-08 15:53:16,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-08 15:53:16,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-08 15:53:16,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-08 15:53:16,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-08 15:53:16,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-08 15:53:16,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-08 15:53:16,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-08 15:53:16,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-08 15:53:16,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-08 15:53:16,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-08 15:53:16,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-08 15:53:16,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-08 15:53:16,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-08 15:53:16,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-08 15:53:16,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-08 15:53:16,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-08 15:53:16,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-08 15:53:16,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-08 15:53:16,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-08 15:53:16,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-08 15:53:16,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-08 15:53:16,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-08 15:53:16,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-08 15:53:16,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-08 15:53:16,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-08 15:53:16,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-08 15:53:16,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-08 15:53:16,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-08 15:53:16,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-08 15:53:16,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-08 15:53:16,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-08 15:53:16,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-08 15:53:16,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-08 15:53:16,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-08 15:53:16,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-08 15:53:16,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-08 15:53:16,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-08 15:53:16,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-08 15:53:16,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-08 15:53:16,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-08 15:53:16,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-08 15:53:16,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-08 15:53:16,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-08 15:53:16,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-08 15:53:16,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-08 15:53:16,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-08 15:53:16,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-08 15:53:16,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-08 15:53:16,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-08 15:53:16,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-08 15:53:16,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-08 15:53:16,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-08 15:53:16,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-08 15:53:16,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-08 15:53:16,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-08 15:53:16,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-08 15:53:16,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-08 15:53:16,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-08 15:53:16,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-08 15:53:16,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-08 15:53:16,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-08 15:53:16,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-08 15:53:16,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-08 15:53:16,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-08 15:53:16,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-08 15:53:16,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-08 15:53:16,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-08 15:53:16,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-08 15:53:16,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-08 15:53:16,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-08 15:53:16,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-08 15:53:16,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-08 15:53:16,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-08 15:53:16,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-08 15:53:16,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-08 15:53:16,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-08 15:53:16,925 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-08 15:53:16,925 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-08 15:53:16,926 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-08 15:53:17,552 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 786 ms to try 5000 different iterations.  Found a solution that moves 4720 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1737655220309171. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0421861739906521, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056891019530807926, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5244444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0421861739906521, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16916666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.233, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-08 15:53:17,569 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv454757238:80 , srv2119853453:82 , srv232931826:82 , srv805778281:82 , srv948806881:82 , srv1078668790:83 , srv128297666:83 , srv1491685040:83 , srv157832699:83 , srv1700565924:83 , srv1713337116:83 , srv1843337773:83 , srv1949605675:83 , srv903623104:83 , srv914662246:83 , srv1154814852:84 , srv1373073360:84 , srv1391623499:84 , srv1427938932:84 , srv1565757425:84 , srv1650519673:84 , srv1730208776:84 , srv1804012614:84 , srv20254933:84 , srv322188390:84 , srv572408198:84 , srv1032840094:85 , srv1306526914:85 , srv1426941501:85 , srv1446740569:85 , srv1492116257:85 , srv1741420780:85 , srv1809566825:85 , srv1865606187:85 , srv2142630400:85 , srv219280996:85 , srv277752129:85 , srv328920534:85 , srv334859030:85 , srv550741197:85 , srv65630548:85 , srv950019898:85 , srv1108208098:86 , srv1249697444:86 , srv1355508237:86 , srv1400527951:86 , srv1753471837:86 , srv1756127224:86 , srv1818126107:86 , srv2016469725:86 , srv2016495454:86 , srv298559727:86 , srv455253557:86 , srv574858268:86 , srv836332868:86 , srv909470651:86 , srv923668394:86 , srv1014595466:87 , srv1127338266:87 , srv1186564537:87 , srv2030182528:87 , srv220167258:87 , srv229363422:87 , srv387586074:87 , srv674603265:87 , srv717987243:87 , srv759026509:87 , srv1374035793:88 , srv1381211730:88 , srv1383364875:88 , srv1412546130:88 , srv1574497805:88 , srv1737966533:88 , srv1893396910:88 , srv2034727245:88 , srv510543993:88 , srv648162819:88 , srv666071834:88 , srv818353502:88 , srv818671758:88 , srv851267417:88 , srv895249072:88 , srv896680763:88 , srv925201202:88 , srv1502649377:89 , srv548611336:89 , srv739939434:89 , srv947988749:89 , srv1228861660:90 , srv1697228349:90 , srv1761851683:90 , srv407735636:90 , srv699556681:90 , srv978625798:90 , srv1124822332:91 , srv2011476985:91 , srv618881394:91 , srv1465039067:92 , srv326780280:96 , srv449688332:465 }
2024-03-08 15:53:17,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-08 15:53:17,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1306526914=11, srv455253557=69, srv2011476985=47, srv277752129=59, srv1383364875=16, srv1154814852=6, srv1032840094=1, srv2119853453=53, srv759026509=83, srv232931826=58, srv1465039067=23, srv2142630400=54, srv925201202=95, srv1809566825=41, srv1391623499=17, srv666071834=78, srv1893396910=45, srv1374035793=14, srv1446740569=22, srv1753471837=37, srv2016469725=48, srv328920534=63, srv896680763=90, srv978625798=99, srv157832699=29, srv1565757425=27, srv895249072=89, srv1426941501=20, srv1186564537=7, srv950019898=98, srv510543993=70, srv914662246=93, srv1843337773=43, srv1249697444=9, srv851267417=88, srv1741420780=36, srv548611336=71, srv699556681=80, srv1804012614=40, srv407735636=66, srv903623104=91, srv1491685040=24, srv229363422=57, srv550741197=72, srv1697228349=31, srv1730208776=34, srv20254933=50, srv2016495454=49, srv387586074=65, srv1700565924=32, srv1400527951=18, srv1737966533=35, srv572408198=73, srv1761851683=39, srv923668394=94, srv1355508237=12, srv454757238=68, srv1228861660=8, srv648162819=76, srv909470651=92, srv1108208098=3, srv449688332=67, srv326780280=62, srv1949605675=46, srv1373073360=13, srv220167258=56, srv574858268=74, srv1574497805=28, srv1127338266=5, srv674603265=79, srv739939434=82, srv818671758=86, srv947988749=96, srv1014595466=0, srv298559727=60, srv322188390=61, srv1078668790=2, srv334859030=64, srv818353502=85, srv1818126107=42, srv1124822332=4, srv717987243=81, srv65630548=77, srv1381211730=15, srv1713337116=33, srv219280996=55, srv2030182528=51, srv618881394=75, srv1865606187=44, srv1492116257=25, srv1650519673=30, srv1427938932=21, srv805778281=84, srv2034727245=52, srv1756127224=38, srv128297666=10, srv948806881=97, srv1412546130=19, srv1502649377=26, srv836332868=87} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-08 15:53:17,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-08 15:53:17,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-08 15:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-08 15:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-08 15:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-08 15:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-08 15:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-08 15:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-08 15:53:17,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-08 15:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-08 15:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-08 15:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-08 15:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-08 15:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-08 15:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-08 15:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-08 15:53:17,606 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-08 15:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-08 15:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-08 15:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-08 15:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-08 15:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-08 15:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-08 15:53:17,607 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-08 15:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-08 15:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-08 15:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-08 15:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-08 15:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-08 15:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-08 15:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-08 15:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-08 15:53:17,608 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-08 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-08 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-08 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-08 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-08 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-08 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-08 15:53:17,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-08 15:53:17,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-08 15:53:17,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-08 15:53:17,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-08 15:53:17,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-08 15:53:17,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-08 15:53:17,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-08 15:53:17,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-08 15:53:17,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-08 15:53:17,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-08 15:53:17,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-08 15:53:17,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-08 15:53:17,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-08 15:53:17,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-08 15:53:17,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-08 15:53:17,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-08 15:53:17,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-08 15:53:17,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-08 15:53:17,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-08 15:53:17,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-08 15:53:17,611 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-08 15:53:17,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-08 15:53:17,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-08 15:53:17,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-08 15:53:17,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-08 15:53:17,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-08 15:53:17,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-08 15:53:17,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-08 15:53:17,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-08 15:53:17,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-08 15:53:17,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-08 15:53:17,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-08 15:53:17,612 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-08 15:53:17,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-08 15:53:17,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-08 15:53:17,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-08 15:53:17,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-08 15:53:17,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-08 15:53:17,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-08 15:53:17,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-08 15:53:17,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-08 15:53:17,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-08 15:53:17,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-08 15:53:17,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-08 15:53:17,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-08 15:53:17,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-08 15:53:17,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-08 15:53:17,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-08 15:53:17,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-08 15:53:17,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-08 15:53:17,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-08 15:53:17,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-08 15:53:17,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-08 15:53:17,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-08 15:53:17,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-08 15:53:17,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-08 15:53:17,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-08 15:53:17,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-08 15:53:17,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-08 15:53:17,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-08 15:53:17,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-08 15:53:17,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-08 15:53:17,780 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0421861739906521, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056891019530807926, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0421861739906521, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16916666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.233, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-08 15:53:17,780 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-08 15:53:17,780 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17373247188452215, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0421861739906521, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056891019530807926, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0421861739906521, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16916666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.233, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-08 15:53:18,072 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 430 ms to try 5000 different iterations.  Found a solution that moves 2554 regions; Going from a computed imbalance of 0.17373247188452215 to a new imbalance of 0.015585882702557865. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022956445257274277); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03559656725653379, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2837777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022956445257274277); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-08 15:53:18,084 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv326780280:77 , srv1124822332:81 , srv1465039067:81 , srv1697228349:81 , srv548611336:81 , srv618881394:81 , srv739939434:81 , srv851267417:81 , srv978625798:81 , srv1374035793:82 , srv1381211730:82 , srv1383364875:82 , srv1502649377:82 , srv1737966533:82 , srv407735636:82 , srv1818126107:83 , srv2011476985:83 , srv510543993:83 , srv666071834:83 , srv674603265:83 , srv925201202:83 , srv1014595466:84 , srv1127338266:84 , srv1249697444:84 , srv1741420780:84 , srv1761851683:84 , srv699556681:84 , srv818353502:84 , srv896680763:84 , srv1108208098:85 , srv1412546130:85 , srv1574497805:85 , srv1809566825:85 , srv2034727245:85 , srv219280996:85 , srv220167258:85 , srv229363422:85 , srv455253557:85 , srv574858268:85 , srv65630548:85 , srv818671758:85 , srv947988749:85 , srv1186564537:86 , srv1306526914:86 , srv1355508237:86 , srv1427938932:86 , srv1565757425:86 , srv1756127224:86 , srv1865606187:86 , srv1893396910:86 , srv2016495454:86 , srv2030182528:86 , srv277752129:86 , srv298559727:86 , srv387586074:86 , srv648162819:86 , srv717987243:86 , srv759026509:86 , srv895249072:86 , srv909470651:86 , srv1391623499:87 , srv1400527951:87 , srv1426941501:87 , srv1730208776:87 , srv1753471837:87 , srv1804012614:87 , srv2016469725:87 , srv20254933:87 , srv334859030:87 , srv572408198:87 , srv903623104:87 , srv923668394:87 , srv1032840094:88 , srv128297666:88 , srv1373073360:88 , srv1446740569:88 , srv1492116257:88 , srv1650519673:88 , srv1949605675:88 , srv2142630400:88 , srv322188390:88 , srv328920534:88 , srv836332868:88 , srv950019898:88 , srv1078668790:89 , srv1154814852:89 , srv1228861660:89 , srv1700565924:89 , srv1843337773:89 , srv550741197:89 , srv805778281:89 , srv914662246:89 , srv1491685040:90 , srv157832699:90 , srv2119853453:90 , srv232931826:90 , srv948806881:91 , srv1713337116:92 , srv454757238:92 , srv449688332:513 }
2024-03-08 15:53:18,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-08 15:53:18,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1306526914=11, srv455253557=69, srv2011476985=47, srv277752129=59, srv1383364875=16, srv1154814852=6, srv1032840094=1, srv2119853453=53, srv759026509=83, srv232931826=58, srv1465039067=23, srv2142630400=54, srv925201202=95, srv1809566825=41, srv1391623499=17, srv666071834=78, srv1893396910=45, srv1374035793=14, srv1446740569=22, srv1753471837=37, srv2016469725=48, srv328920534=63, srv896680763=90, srv978625798=99, srv157832699=29, srv1565757425=27, srv895249072=89, srv1426941501=20, srv1186564537=7, srv950019898=98, srv510543993=70, srv914662246=93, srv1843337773=43, srv1249697444=9, srv851267417=88, srv1741420780=36, srv548611336=71, srv699556681=80, srv1804012614=40, srv407735636=66, srv903623104=91, srv1491685040=24, srv229363422=57, srv550741197=72, srv1697228349=31, srv1730208776=34, srv20254933=50, srv2016495454=49, srv387586074=65, srv1700565924=32, srv1400527951=18, srv1737966533=35, srv572408198=73, srv1761851683=39, srv923668394=94, srv1355508237=12, srv454757238=68, srv1228861660=8, srv648162819=76, srv909470651=92, srv1108208098=3, srv449688332=67, srv326780280=62, srv1949605675=46, srv1373073360=13, srv220167258=56, srv574858268=74, srv1574497805=28, srv1127338266=5, srv674603265=79, srv739939434=82, srv818671758=86, srv947988749=96, srv1014595466=0, srv298559727=60, srv322188390=61, srv1078668790=2, srv334859030=64, srv818353502=85, srv1818126107=42, srv1124822332=4, srv717987243=81, srv65630548=77, srv1381211730=15, srv1713337116=33, srv219280996=55, srv2030182528=51, srv618881394=75, srv1865606187=44, srv1492116257=25, srv1650519673=30, srv1427938932=21, srv805778281=84, srv2034727245=52, srv1756127224=38, srv128297666=10, srv948806881=97, srv1412546130=19, srv1502649377=26, srv836332868=87} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-08 15:53:18,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-08 15:53:18,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-08 15:53:18,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-08 15:53:18,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-08 15:53:18,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-08 15:53:18,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-08 15:53:18,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-08 15:53:18,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-08 15:53:18,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-08 15:53:18,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-08 15:53:18,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-08 15:53:18,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-08 15:53:18,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-08 15:53:18,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-08 15:53:18,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-08 15:53:18,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-08 15:53:18,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-08 15:53:18,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-08 15:53:18,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-08 15:53:18,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-08 15:53:18,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-08 15:53:18,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-08 15:53:18,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-08 15:53:18,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-08 15:53:18,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-08 15:53:18,113 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-08 15:53:18,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-08 15:53:18,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-08 15:53:18,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-08 15:53:18,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-08 15:53:18,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-08 15:53:18,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-08 15:53:18,114 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-08 15:53:18,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-08 15:53:18,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-08 15:53:18,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-08 15:53:18,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-08 15:53:18,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-08 15:53:18,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-08 15:53:18,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-08 15:53:18,115 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-08 15:53:18,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-08 15:53:18,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-08 15:53:18,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-08 15:53:18,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-08 15:53:18,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-08 15:53:18,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-08 15:53:18,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-08 15:53:18,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-08 15:53:18,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-08 15:53:18,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-08 15:53:18,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-08 15:53:18,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-08 15:53:18,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-08 15:53:18,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-08 15:53:18,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-08 15:53:18,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-08 15:53:18,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-08 15:53:18,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-08 15:53:18,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-08 15:53:18,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-08 15:53:18,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-08 15:53:18,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-08 15:53:18,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-08 15:53:18,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-08 15:53:18,118 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-08 15:53:18,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-08 15:53:18,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-08 15:53:18,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-08 15:53:18,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-08 15:53:18,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-08 15:53:18,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-08 15:53:18,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-08 15:53:18,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-08 15:53:18,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-08 15:53:18,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-08 15:53:18,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-08 15:53:18,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-08 15:53:18,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-08 15:53:18,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-08 15:53:18,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-08 15:53:18,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-08 15:53:18,120 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-08 15:53:18,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-08 15:53:18,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-08 15:53:18,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-08 15:53:18,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-08 15:53:18,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-08 15:53:18,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-08 15:53:18,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-08 15:53:18,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-08 15:53:18,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-08 15:53:18,121 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-08 15:53:18,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-08 15:53:18,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-08 15:53:18,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-08 15:53:18,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-08 15:53:18,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-08 15:53:18,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-08 15:53:18,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-08 15:53:18,122 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-08 15:53:18,253 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022956445257274277); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03559656725653379, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022956445257274277); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-08 15:53:18,253 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-08 15:53:18,253 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01556799921232636, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022956445257274277); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03559656725653379, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022956445257274277); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012583333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-08 15:53:18,643 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 497 ms to try 5000 different iterations.  Found a solution that moves 1010 regions; Going from a computed imbalance of 0.01556799921232636 to a new imbalance of 6.911112605699597E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006649810339649611); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006513389472789296); MoveCostFunction : (multiplier=7.0, imbalance=0.11222222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006649810339649611); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-08 15:53:18,646 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1228861660:79 , srv1713337116:83 , srv2011476985:83 , srv699556681:83 , srv836332868:83 , srv895249072:83 , srv1186564537:84 , srv1400527951:84 , srv1465039067:84 , srv1492116257:84 , srv1761851683:84 , srv1893396910:84 , srv2034727245:84 , srv2142630400:84 , srv326780280:84 , srv328920534:84 , srv648162819:84 , srv759026509:84 , srv818671758:84 , srv947988749:84 , srv950019898:84 , srv1032840094:85 , srv1078668790:85 , srv1124822332:85 , srv1154814852:85 , srv1412546130:85 , srv1446740569:85 , srv1491685040:85 , srv1574497805:85 , srv157832699:85 , srv1650519673:85 , srv1700565924:85 , srv1753471837:85 , srv1756127224:85 , srv1843337773:85 , srv2016469725:85 , srv2016495454:85 , srv2030182528:85 , srv220167258:85 , srv229363422:85 , srv232931826:85 , srv298559727:85 , srv387586074:85 , srv407735636:85 , srv454757238:85 , srv550741197:85 , srv574858268:85 , srv717987243:85 , srv818353502:85 , srv896680763:85 , srv909470651:85 , srv923668394:85 , srv948806881:85 , srv1014595466:86 , srv1108208098:86 , srv1127338266:86 , srv128297666:86 , srv1306526914:86 , srv1355508237:86 , srv1373073360:86 , srv1391623499:86 , srv1426941501:86 , srv1502649377:86 , srv1697228349:86 , srv1730208776:86 , srv1865606187:86 , srv1949605675:86 , srv20254933:86 , srv277752129:86 , srv322188390:86 , srv334859030:86 , srv510543993:86 , srv572408198:86 , srv618881394:86 , srv914662246:86 , srv925201202:86 , srv978625798:86 , srv1374035793:87 , srv1381211730:87 , srv1383364875:87 , srv1565757425:87 , srv1737966533:87 , srv1804012614:87 , srv1809566825:87 , srv2119853453:87 , srv455253557:87 , srv548611336:87 , srv666071834:87 , srv674603265:87 , srv739939434:87 , srv805778281:87 , srv903623104:87 , srv1249697444:88 , srv1427938932:88 , srv1818126107:88 , srv219280996:88 , srv65630548:88 , srv1741420780:89 , srv851267417:89 , srv449688332:539 }
2024-03-08 15:53:18,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-08 15:53:18,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1306526914=11, srv455253557=69, srv2011476985=47, srv277752129=59, srv1383364875=16, srv1154814852=6, srv1032840094=1, srv2119853453=53, srv759026509=83, srv232931826=58, srv1465039067=23, srv2142630400=54, srv925201202=95, srv1809566825=41, srv1391623499=17, srv666071834=78, srv1893396910=45, srv1374035793=14, srv1446740569=22, srv1753471837=37, srv2016469725=48, srv328920534=63, srv896680763=90, srv978625798=99, srv157832699=29, srv1565757425=27, srv895249072=89, srv1426941501=20, srv1186564537=7, srv950019898=98, srv510543993=70, srv914662246=93, srv1843337773=43, srv1249697444=9, srv851267417=88, srv1741420780=36, srv548611336=71, srv699556681=80, srv1804012614=40, srv407735636=66, srv903623104=91, srv1491685040=24, srv229363422=57, srv550741197=72, srv1697228349=31, srv1730208776=34, srv20254933=50, srv2016495454=49, srv387586074=65, srv1700565924=32, srv1400527951=18, srv1737966533=35, srv572408198=73, srv1761851683=39, srv923668394=94, srv1355508237=12, srv454757238=68, srv1228861660=8, srv648162819=76, srv909470651=92, srv1108208098=3, srv449688332=67, srv326780280=62, srv1949605675=46, srv1373073360=13, srv220167258=56, srv574858268=74, srv1574497805=28, srv1127338266=5, srv674603265=79, srv739939434=82, srv818671758=86, srv947988749=96, srv1014595466=0, srv298559727=60, srv322188390=61, srv1078668790=2, srv334859030=64, srv818353502=85, srv1818126107=42, srv1124822332=4, srv717987243=81, srv65630548=77, srv1381211730=15, srv1713337116=33, srv219280996=55, srv2030182528=51, srv618881394=75, srv1865606187=44, srv1492116257=25, srv1650519673=30, srv1427938932=21, srv805778281=84, srv2034727245=52, srv1756127224=38, srv128297666=10, srv948806881=97, srv1412546130=19, srv1502649377=26, srv836332868=87} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-08 15:53:18,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-08 15:53:18,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-08 15:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-08 15:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-08 15:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-08 15:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-08 15:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-08 15:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-08 15:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-08 15:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-08 15:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-08 15:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-08 15:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-08 15:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-08 15:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-08 15:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-08 15:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-08 15:53:18,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-08 15:53:18,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-08 15:53:18,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-08 15:53:18,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-08 15:53:18,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-08 15:53:18,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-08 15:53:18,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-08 15:53:18,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-08 15:53:18,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-08 15:53:18,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-08 15:53:18,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-08 15:53:18,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-08 15:53:18,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-08 15:53:18,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-08 15:53:18,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-08 15:53:18,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-08 15:53:18,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-08 15:53:18,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-08 15:53:18,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-08 15:53:18,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-08 15:53:18,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-08 15:53:18,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-08 15:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-08 15:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-08 15:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-08 15:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-08 15:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-08 15:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-08 15:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-08 15:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-08 15:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-08 15:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-08 15:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-08 15:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-08 15:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-08 15:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-08 15:53:18,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-08 15:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-08 15:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-08 15:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-08 15:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-08 15:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-08 15:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-08 15:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-08 15:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-08 15:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-08 15:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-08 15:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-08 15:53:18,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-08 15:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-08 15:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-08 15:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-08 15:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-08 15:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-08 15:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-08 15:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-08 15:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-08 15:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-08 15:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-08 15:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-08 15:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-08 15:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-08 15:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-08 15:53:18,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-08 15:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-08 15:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-08 15:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-08 15:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-08 15:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-08 15:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-08 15:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-08 15:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-08 15:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-08 15:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-08 15:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-08 15:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-08 15:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-08 15:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-08 15:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-08 15:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-08 15:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-08 15:53:18,671 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-08 15:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-08 15:53:18,672 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-08 15:53:18,805 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006649810339649611); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006513389472789296); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006649810339649611); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-08 15:53:18,805 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-08 15:53:18,806 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.840390894134147E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006649810339649611); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006513389472789296); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006649810339649611); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-08 15:53:19,091 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 411 ms to try 5000 different iterations.  Found a solution that moves 277 regions; Going from a computed imbalance of 6.840390894134147E-4 to a new imbalance of 2.1967758288026444E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.03077777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-08 15:53:19,092 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1412546130:83 , srv548611336:83 , srv550741197:83 , srv1108208098:84 , srv1154814852:84 , srv1249697444:84 , srv1355508237:84 , srv1373073360:84 , srv1400527951:84 , srv1426941501:84 , srv1427938932:84 , srv1446740569:84 , srv1491685040:84 , srv157832699:84 , srv1713337116:84 , srv1737966533:84 , srv1741420780:84 , srv1753471837:84 , srv1761851683:84 , srv1804012614:84 , srv1893396910:84 , srv2011476985:84 , srv2016469725:84 , srv2030182528:84 , srv2119853453:84 , srv219280996:84 , srv326780280:84 , srv334859030:84 , srv387586074:84 , srv455253557:84 , srv618881394:84 , srv65630548:84 , srv666071834:84 , srv717987243:84 , srv805778281:84 , srv896680763:84 , srv903623104:84 , srv914662246:84 , srv923668394:84 , srv947988749:84 , srv1014595466:85 , srv1032840094:85 , srv1078668790:85 , srv1124822332:85 , srv1127338266:85 , srv1186564537:85 , srv1228861660:85 , srv1306526914:85 , srv1374035793:85 , srv1381211730:85 , srv1383364875:85 , srv1391623499:85 , srv1465039067:85 , srv1492116257:85 , srv1502649377:85 , srv1565757425:85 , srv1574497805:85 , srv1650519673:85 , srv1697228349:85 , srv1700565924:85 , srv1730208776:85 , srv1756127224:85 , srv1809566825:85 , srv1818126107:85 , srv1843337773:85 , srv1865606187:85 , srv1949605675:85 , srv2016495454:85 , srv20254933:85 , srv2034727245:85 , srv2142630400:85 , srv220167258:85 , srv229363422:85 , srv232931826:85 , srv277752129:85 , srv298559727:85 , srv322188390:85 , srv328920534:85 , srv407735636:85 , srv454757238:85 , srv510543993:85 , srv572408198:85 , srv648162819:85 , srv674603265:85 , srv699556681:85 , srv739939434:85 , srv759026509:85 , srv818353502:85 , srv818671758:85 , srv836332868:85 , srv851267417:85 , srv895249072:85 , srv909470651:85 , srv925201202:85 , srv948806881:85 , srv950019898:85 , srv978625798:85 , srv128297666:86 , srv574858268:87 , srv449688332:625 }
2024-03-08 15:53:19,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-08 15:53:19,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1306526914=11, srv455253557=69, srv2011476985=47, srv277752129=59, srv1383364875=16, srv1154814852=6, srv1032840094=1, srv2119853453=53, srv759026509=83, srv232931826=58, srv1465039067=23, srv2142630400=54, srv925201202=95, srv1809566825=41, srv1391623499=17, srv666071834=78, srv1893396910=45, srv1374035793=14, srv1446740569=22, srv1753471837=37, srv2016469725=48, srv328920534=63, srv896680763=90, srv978625798=99, srv157832699=29, srv1565757425=27, srv895249072=89, srv1426941501=20, srv1186564537=7, srv950019898=98, srv510543993=70, srv914662246=93, srv1843337773=43, srv1249697444=9, srv851267417=88, srv1741420780=36, srv548611336=71, srv699556681=80, srv1804012614=40, srv407735636=66, srv903623104=91, srv1491685040=24, srv229363422=57, srv550741197=72, srv1697228349=31, srv1730208776=34, srv20254933=50, srv2016495454=49, srv387586074=65, srv1700565924=32, srv1400527951=18, srv1737966533=35, srv572408198=73, srv1761851683=39, srv923668394=94, srv1355508237=12, srv454757238=68, srv1228861660=8, srv648162819=76, srv909470651=92, srv1108208098=3, srv449688332=67, srv326780280=62, srv1949605675=46, srv1373073360=13, srv220167258=56, srv574858268=74, srv1574497805=28, srv1127338266=5, srv674603265=79, srv739939434=82, srv818671758=86, srv947988749=96, srv1014595466=0, srv298559727=60, srv322188390=61, srv1078668790=2, srv334859030=64, srv818353502=85, srv1818126107=42, srv1124822332=4, srv717987243=81, srv65630548=77, srv1381211730=15, srv1713337116=33, srv219280996=55, srv2030182528=51, srv618881394=75, srv1865606187=44, srv1492116257=25, srv1650519673=30, srv1427938932=21, srv805778281=84, srv2034727245=52, srv1756127224=38, srv128297666=10, srv948806881=97, srv1412546130=19, srv1502649377=26, srv836332868=87} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-08 15:53:19,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-08 15:53:19,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-08 15:53:19,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-08 15:53:19,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-08 15:53:19,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-08 15:53:19,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-08 15:53:19,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-08 15:53:19,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-08 15:53:19,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-08 15:53:19,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-08 15:53:19,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-08 15:53:19,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-08 15:53:19,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-08 15:53:19,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-08 15:53:19,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-08 15:53:19,107 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-08 15:53:19,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-08 15:53:19,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-08 15:53:19,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-08 15:53:19,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-08 15:53:19,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-08 15:53:19,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-08 15:53:19,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-08 15:53:19,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-08 15:53:19,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-08 15:53:19,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-08 15:53:19,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-08 15:53:19,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-08 15:53:19,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-08 15:53:19,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-08 15:53:19,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-08 15:53:19,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-08 15:53:19,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-08 15:53:19,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-08 15:53:19,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-08 15:53:19,108 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-08 15:53:19,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-08 15:53:19,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-08 15:53:19,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-08 15:53:19,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-08 15:53:19,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-08 15:53:19,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-08 15:53:19,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-08 15:53:19,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-08 15:53:19,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-08 15:53:19,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-08 15:53:19,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-08 15:53:19,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-08 15:53:19,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-08 15:53:19,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-08 15:53:19,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-08 15:53:19,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-08 15:53:19,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-08 15:53:19,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-08 15:53:19,109 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-08 15:53:19,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-08 15:53:19,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-08 15:53:19,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-08 15:53:19,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-08 15:53:19,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-08 15:53:19,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-08 15:53:19,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-08 15:53:19,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-08 15:53:19,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-08 15:53:19,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-08 15:53:19,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-08 15:53:19,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-08 15:53:19,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-08 15:53:19,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-08 15:53:19,110 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-08 15:53:19,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-08 15:53:19,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-08 15:53:19,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-08 15:53:19,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-08 15:53:19,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-08 15:53:19,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-08 15:53:19,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-08 15:53:19,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-08 15:53:19,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-08 15:53:19,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-08 15:53:19,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-08 15:53:19,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-08 15:53:19,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-08 15:53:19,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-08 15:53:19,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-08 15:53:19,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-08 15:53:19,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-08 15:53:19,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-08 15:53:19,111 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-08 15:53:19,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-08 15:53:19,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-08 15:53:19,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-08 15:53:19,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-08 15:53:19,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-08 15:53:19,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-08 15:53:19,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-08 15:53:19,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-08 15:53:19,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-08 15:53:19,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-08 15:53:19,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-08 15:53:19,112 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-08 15:53:19,216 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=2.002816283222151E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-08 15:53:19,216 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1412546130:83 , srv548611336:83 , srv550741197:83 , srv1108208098:84 , srv1154814852:84 , srv1249697444:84 , srv1355508237:84 , srv1373073360:84 , srv1400527951:84 , srv1426941501:84 , srv1427938932:84 , srv1446740569:84 , srv1491685040:84 , srv157832699:84 , srv1713337116:84 , srv1737966533:84 , srv1741420780:84 , srv1753471837:84 , srv1761851683:84 , srv1804012614:84 , srv1893396910:84 , srv2011476985:84 , srv2016469725:84 , srv2030182528:84 , srv2119853453:84 , srv219280996:84 , srv326780280:84 , srv334859030:84 , srv387586074:84 , srv455253557:84 , srv618881394:84 , srv65630548:84 , srv666071834:84 , srv717987243:84 , srv805778281:84 , srv896680763:84 , srv903623104:84 , srv914662246:84 , srv923668394:84 , srv947988749:84 , srv1014595466:85 , srv1032840094:85 , srv1078668790:85 , srv1124822332:85 , srv1127338266:85 , srv1186564537:85 , srv1228861660:85 , srv1306526914:85 , srv1374035793:85 , srv1381211730:85 , srv1383364875:85 , srv1391623499:85 , srv1465039067:85 , srv1492116257:85 , srv1502649377:85 , srv1565757425:85 , srv1574497805:85 , srv1650519673:85 , srv1697228349:85 , srv1700565924:85 , srv1730208776:85 , srv1756127224:85 , srv1809566825:85 , srv1818126107:85 , srv1843337773:85 , srv1865606187:85 , srv1949605675:85 , srv2016495454:85 , srv20254933:85 , srv2034727245:85 , srv2142630400:85 , srv220167258:85 , srv229363422:85 , srv232931826:85 , srv277752129:85 , srv298559727:85 , srv322188390:85 , srv328920534:85 , srv407735636:85 , srv454757238:85 , srv510543993:85 , srv572408198:85 , srv648162819:85 , srv674603265:85 , srv699556681:85 , srv739939434:85 , srv759026509:85 , srv818353502:85 , srv818671758:85 , srv836332868:85 , srv851267417:85 , srv895249072:85 , srv909470651:85 , srv925201202:85 , srv948806881:85 , srv950019898:85 , srv978625798:85 , srv128297666:86 , srv574858268:87 , srv449688332:625 }
2024-03-08 15:53:19,241 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=271 (was 250) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=9189 (was 9926)
2024-03-08 15:53:19,250 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=271, ProcessCount=170, AvailableMemoryMB=9191
2024-03-08 15:53:19,251 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-08 15:53:19,251 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-03-08 15:53:19,261 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1106311830:84 , srv1168863621:84 , srv1218762725:84 , srv1352362588:84 , srv1397277309:84 , srv1400226677:84 , srv1452086638:84 , srv1622742339:84 , srv1674729512:84 , srv1720055980:84 , srv1774695538:84 , srv1799353877:84 , srv1837596217:84 , srv1947242867:84 , srv1982486180:84 , srv201117999:84 , srv2033096147:84 , srv2083633799:84 , srv2132723908:84 , srv249205762:84 , srv27946747:84 , srv559510056:84 , srv6894398:84 , srv706513691:84 , srv747131911:84 , srv833491665:84 , srv855178934:84 , srv975749672:84 , srv985762284:84 , srv1841110977:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-03-08 15:53:19,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-08 15:53:19,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1622742339=7, srv1982486180=15, srv2132723908=19, srv1218762725=2, srv1720055980=9, srv706513691=24, srv2083633799=18, srv1947242867=14, srv833491665=26, srv975749672=28, srv249205762=20, srv1106311830=0, srv1352362588=3, srv1400226677=5, srv1452086638=6, srv1837596217=12, srv6894398=23, srv747131911=25, srv1774695538=10, srv985762284=29, srv1674729512=8, srv201117999=16, srv1799353877=11, srv2033096147=17, srv855178934=27, srv1841110977=13, srv559510056=22, srv27946747=21, srv1397277309=4, srv1168863621=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-08 15:53:19,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-08 15:53:19,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-08 15:53:19,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-08 15:53:19,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-08 15:53:19,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-08 15:53:19,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-08 15:53:19,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-08 15:53:19,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-08 15:53:19,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-08 15:53:19,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-08 15:53:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-08 15:53:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-08 15:53:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-08 15:53:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-08 15:53:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-08 15:53:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-08 15:53:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-08 15:53:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-08 15:53:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-08 15:53:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-08 15:53:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-08 15:53:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-08 15:53:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-08 15:53:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-08 15:53:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-08 15:53:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-08 15:53:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-08 15:53:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-08 15:53:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-08 15:53:19,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-08 15:53:19,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-08 15:53:19,288 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0804074627238416, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-08 15:53:19,288 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912446404553777, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0804074627238416, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-03-08 15:55:19,276 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5682613 different iterations.  Found a solution that moves 1859 regions; Going from a computed imbalance of 0.9912446404553777 to a new imbalance of 4.338998739279625E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6885185185185185, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-08 15:55:19,282 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1106311830:90 , srv1168863621:90 , srv1218762725:90 , srv1352362588:90 , srv1397277309:90 , srv1400226677:90 , srv1452086638:90 , srv1622742339:90 , srv1674729512:90 , srv1720055980:90 , srv1774695538:90 , srv1799353877:90 , srv1837596217:90 , srv1841110977:90 , srv1947242867:90 , srv1982486180:90 , srv201117999:90 , srv2033096147:90 , srv2083633799:90 , srv2132723908:90 , srv249205762:90 , srv27946747:90 , srv559510056:90 , srv6894398:90 , srv706513691:90 , srv747131911:90 , srv833491665:90 , srv855178934:90 , srv975749672:90 , srv985762284:90 }
2024-03-08 15:55:19,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-08 15:55:19,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1622742339=7, srv1982486180=15, srv2132723908=19, srv1218762725=2, srv1720055980=9, srv706513691=24, srv2083633799=18, srv1947242867=14, srv833491665=26, srv975749672=28, srv249205762=20, srv1106311830=0, srv1352362588=3, srv1400226677=5, srv1452086638=6, srv1837596217=12, srv6894398=23, srv747131911=25, srv1774695538=10, srv985762284=29, srv1674729512=8, srv201117999=16, srv1799353877=11, srv2033096147=17, srv855178934=27, srv1841110977=13, srv559510056=22, srv27946747=21, srv1397277309=4, srv1168863621=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-08 15:55:19,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-08 15:55:19,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-08 15:55:19,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-08 15:55:19,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-08 15:55:19,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-08 15:55:19,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-08 15:55:19,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-08 15:55:19,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-08 15:55:19,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-08 15:55:19,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-08 15:55:19,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-08 15:55:19,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-08 15:55:19,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-08 15:55:19,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-08 15:55:19,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-08 15:55:19,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-08 15:55:19,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-08 15:55:19,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-08 15:55:19,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-08 15:55:19,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-08 15:55:19,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-08 15:55:19,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-08 15:55:19,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-08 15:55:19,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-08 15:55:19,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-08 15:55:19,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-08 15:55:19,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-08 15:55:19,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-08 15:55:19,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-08 15:55:19,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-08 15:55:19,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-08 15:55:19,300 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-08 15:55:19,300 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1106311830:90 , srv1168863621:90 , srv1218762725:90 , srv1352362588:90 , srv1397277309:90 , srv1400226677:90 , srv1452086638:90 , srv1622742339:90 , srv1674729512:90 , srv1720055980:90 , srv1774695538:90 , srv1799353877:90 , srv1837596217:90 , srv1841110977:90 , srv1947242867:90 , srv1982486180:90 , srv201117999:90 , srv2033096147:90 , srv2083633799:90 , srv2132723908:90 , srv249205762:90 , srv27946747:90 , srv559510056:90 , srv6894398:90 , srv706513691:90 , srv747131911:90 , srv833491665:90 , srv855178934:90 , srv975749672:90 , srv985762284:90 }
2024-03-08 15:55:19,314 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=153 (was 271), ProcessCount=168 (was 170), AvailableMemoryMB=8425 (was 9191)