2024-02-24 11:53:20,054 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-02-24 11:53:20,376 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-02-24 11:53:20,502 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-02-24 11:53:20,568 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-02-24 11:53:20,576 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-02-24 11:53:20,592 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=272, ProcessCount=170, AvailableMemoryMB=22077
2024-02-24 11:53:20,594 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-02-24 11:53:20,594 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-02-24 11:53:20,739 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1026452051:84 , srv1027537536:84 , srv1041680745:84 , srv1044309165:84 , srv1045514077:84 , srv1047012532:84 , srv1100318467:84 , srv1120011215:84 , srv1124849205:84 , srv1162451994:84 , srv1163308610:84 , srv1179118675:84 , srv1188045530:84 , srv1218084619:84 , srv122738517:84 , srv1256894853:84 , srv1261585312:84 , srv1273002670:84 , srv1289087975:84 , srv1301606746:84 , srv1316083718:84 , srv1327187128:84 , srv1354223332:84 , srv1356468339:84 , srv1361928933:84 , srv1370971554:84 , srv1390273947:84 , srv139615934:84 , srv1424432622:84 , srv1430396028:84 , srv1450411964:84 , srv1495253974:84 , srv1569638132:84 , srv1576509943:84 , srv1578522914:84 , srv1580558849:84 , srv1593023053:84 , srv1594888723:84 , srv1601069010:84 , srv1627650727:84 , srv1629137915:84 , srv1648040864:84 , srv1650568028:84 , srv1708209537:84 , srv174018332:84 , srv1762554429:84 , srv1772812900:84 , srv1800392445:84 , srv1804606025:84 , srv1805250207:84 , srv1843395196:84 , srv1865037287:84 , srv1869094201:84 , srv1887745879:84 , srv1902479205:84 , srv1945350098:84 , srv1962315526:84 , srv1976911852:84 , srv1980024641:84 , srv1993405595:84 , srv2012375239:84 , srv2034597569:84 , srv2063678358:84 , srv218118297:84 , srv241693789:84 , srv246969525:84 , srv262201298:84 , srv284463086:84 , srv297199555:84 , srv403122589:84 , srv405277231:84 , srv436299316:84 , srv458790993:84 , srv544818308:84 , srv560100299:84 , srv570706431:84 , srv577489820:84 , srv58651320:84 , srv61780420:84 , srv625192601:84 , srv645612092:84 , srv658598863:84 , srv668278123:84 , srv66830231:84 , srv688793622:84 , srv716601816:84 , srv739332935:84 , srv806014939:84 , srv826579456:84 , srv836322980:84 , srv836720476:84 , srv89266318:84 , srv922137:84 , srv957097289:84 , srv978136603:84 , srv981724973:84 , srv988777985:84 , srv997777786:84 , srv999017102:84 , srv1260002150:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-02-24 11:53:20,751 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-24 11:53:20,760 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1980024641=59, srv1044309165=3, srv1580558849=36, srv2034597569=62, srv218118297=64, srv1576509943=34, srv1800392445=48, srv922137=93, srv1256894853=15, srv981724973=96, srv1993405595=60, srv836720476=91, srv1865037287=52, srv284463086=68, srv405277231=71, srv668278123=83, srv957097289=94, srv174018332=45, srv806014939=88, srv1869094201=53, srv1424432622=29, srv1354223332=23, srv1027537536=1, srv1218084619=13, srv1772812900=47, srv997777786=98, srv458790993=73, srv560100299=75, srv988777985=97, srv2012375239=61, srv1650568028=43, srv1124849205=8, srv1601069010=39, srv1450411964=31, srv688793622=85, srv1361928933=25, srv1356468339=24, srv1370971554=26, srv122738517=14, srv1188045530=12, srv1569638132=33, srv836322980=90, srv1163308610=10, srv58651320=78, srv1301606746=20, srv1805250207=50, srv739332935=87, srv577489820=77, srv262201298=67, srv1327187128=22, srv716601816=86, srv1902479205=55, srv1026452051=0, srv658598863=82, srv1430396028=30, srv1648040864=42, srv1316083718=21, srv1495253974=32, srv61780420=79, srv1289087975=19, srv1390273947=27, srv241693789=65, srv1762554429=46, srv66830231=84, srv826579456=89, srv246969525=66, srv1179118675=11, srv2063678358=63, srv436299316=72, srv1629137915=41, srv1962315526=57, srv1260002150=16, srv1100318467=6, srv1976911852=58, srv645612092=81, srv1945350098=56, srv89266318=92, srv1261585312=17, srv999017102=99, srv297199555=69, srv1887745879=54, srv1804606025=49, srv1627650727=40, srv978136603=95, srv1047012532=5, srv544818308=74, srv1273002670=18, srv1578522914=35, srv570706431=76, srv625192601=80, srv1045514077=4, srv139615934=28, srv1708209537=44, srv1162451994=9, srv1593023053=37, srv403122589=70, srv1594888723=38, srv1041680745=2, srv1120011215=7, srv1843395196=51} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-24 11:53:20,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-24 11:53:20,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-24 11:53:20,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-24 11:53:20,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-24 11:53:20,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-24 11:53:20,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-24 11:53:20,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-24 11:53:20,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-24 11:53:20,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-24 11:53:20,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-24 11:53:20,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-24 11:53:20,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-24 11:53:20,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-24 11:53:20,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-24 11:53:20,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-24 11:53:20,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-24 11:53:20,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-24 11:53:20,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-24 11:53:20,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-24 11:53:20,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-24 11:53:20,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-24 11:53:20,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-24 11:53:20,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-24 11:53:20,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-24 11:53:20,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-24 11:53:20,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-24 11:53:20,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-24 11:53:20,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-24 11:53:20,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-24 11:53:20,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-24 11:53:20,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-24 11:53:20,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-24 11:53:20,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-24 11:53:20,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-24 11:53:20,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-24 11:53:20,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-24 11:53:20,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-24 11:53:20,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-24 11:53:20,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-24 11:53:20,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-24 11:53:20,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-24 11:53:20,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-24 11:53:20,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-24 11:53:20,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-24 11:53:20,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-24 11:53:20,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-24 11:53:20,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-24 11:53:20,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-24 11:53:20,844 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-24 11:53:20,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-24 11:53:20,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-24 11:53:20,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-24 11:53:20,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-24 11:53:20,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-24 11:53:20,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-24 11:53:20,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-24 11:53:20,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-24 11:53:20,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-24 11:53:20,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-24 11:53:20,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-24 11:53:20,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-24 11:53:20,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-24 11:53:20,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-24 11:53:20,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-24 11:53:20,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-24 11:53:20,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-24 11:53:20,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-24 11:53:20,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-24 11:53:20,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-24 11:53:20,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-24 11:53:20,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-24 11:53:20,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-24 11:53:20,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-24 11:53:20,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-24 11:53:20,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-24 11:53:20,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-24 11:53:20,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-24 11:53:20,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-24 11:53:20,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-24 11:53:20,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-24 11:53:20,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-24 11:53:20,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-24 11:53:20,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-24 11:53:20,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-24 11:53:20,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-24 11:53:20,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-24 11:53:20,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-24 11:53:20,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-24 11:53:20,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-24 11:53:20,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-24 11:53:20,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-24 11:53:20,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-24 11:53:20,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-24 11:53:20,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-24 11:53:20,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-24 11:53:20,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-24 11:53:20,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-24 11:53:20,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-24 11:53:20,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-24 11:53:20,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-24 11:53:20,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-24 11:53:21,031 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 11:53:21,031 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-24 11:53:21,032 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-24 11:53:21,461 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 588 ms to try 5000 different iterations.  Found a solution that moves 4710 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17701754049421348. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039586338417444106, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05547064217817214, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5233333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039586338417444106, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17216666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23933333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 11:53:21,483 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1273002670:81 , srv957097289:82 , srv1762554429:83 , srv1902479205:83 , srv1993405595:83 , srv2034597569:83 , srv66830231:83 , srv988777985:83 , srv1120011215:84 , srv1188045530:84 , srv122738517:84 , srv1256894853:84 , srv1301606746:84 , srv1390273947:84 , srv1576509943:84 , srv1708209537:84 , srv174018332:84 , srv1976911852:84 , srv262201298:84 , srv403122589:84 , srv89266318:84 , srv978136603:84 , srv997777786:84 , srv1163308610:85 , srv1361928933:85 , srv1430396028:85 , srv1805250207:85 , srv1869094201:85 , srv1887745879:85 , srv826579456:85 , srv836720476:85 , srv922137:85 , srv999017102:85 , srv1027537536:86 , srv1047012532:86 , srv1356468339:86 , srv1370971554:86 , srv1569638132:86 , srv1594888723:86 , srv1601069010:86 , srv1648040864:86 , srv1772812900:86 , srv1843395196:86 , srv1865037287:86 , srv1945350098:86 , srv2012375239:86 , srv284463086:86 , srv405277231:86 , srv544818308:86 , srv688793622:86 , srv1026452051:87 , srv1041680745:87 , srv1044309165:87 , srv1100318467:87 , srv1316083718:87 , srv1327187128:87 , srv1450411964:87 , srv1650568028:87 , srv1804606025:87 , srv1962315526:87 , srv241693789:87 , srv246969525:87 , srv560100299:87 , srv58651320:87 , srv61780420:87 , srv645612092:87 , srv658598863:87 , srv668278123:87 , srv716601816:87 , srv739332935:87 , srv836322980:87 , srv981724973:87 , srv1045514077:88 , srv1218084619:88 , srv139615934:88 , srv1580558849:88 , srv1980024641:88 , srv2063678358:88 , srv218118297:88 , srv436299316:88 , srv458790993:88 , srv1124849205:89 , srv1261585312:89 , srv1289087975:89 , srv1354223332:89 , srv1495253974:89 , srv1627650727:89 , srv1629137915:89 , srv297199555:89 , srv806014939:89 , srv1179118675:90 , srv1578522914:90 , srv1593023053:90 , srv570706431:90 , srv1162451994:91 , srv1424432622:91 , srv1800392445:91 , srv625192601:92 , srv577489820:93 , srv1260002150:442 }
2024-02-24 11:53:21,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-24 11:53:21,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1980024641=59, srv1044309165=3, srv1580558849=36, srv2034597569=62, srv218118297=64, srv1576509943=34, srv1800392445=48, srv922137=93, srv1256894853=15, srv981724973=96, srv1993405595=60, srv836720476=91, srv1865037287=52, srv284463086=68, srv405277231=71, srv668278123=83, srv957097289=94, srv174018332=45, srv806014939=88, srv1869094201=53, srv1424432622=29, srv1354223332=23, srv1027537536=1, srv1218084619=13, srv1772812900=47, srv997777786=98, srv458790993=73, srv560100299=75, srv988777985=97, srv2012375239=61, srv1650568028=43, srv1124849205=8, srv1601069010=39, srv1450411964=31, srv688793622=85, srv1361928933=25, srv1356468339=24, srv1370971554=26, srv122738517=14, srv1188045530=12, srv1569638132=33, srv836322980=90, srv1163308610=10, srv58651320=78, srv1301606746=20, srv1805250207=50, srv739332935=87, srv577489820=77, srv262201298=67, srv1327187128=22, srv716601816=86, srv1902479205=55, srv1026452051=0, srv658598863=82, srv1430396028=30, srv1648040864=42, srv1316083718=21, srv1495253974=32, srv61780420=79, srv1289087975=19, srv1390273947=27, srv241693789=65, srv1762554429=46, srv66830231=84, srv826579456=89, srv246969525=66, srv1179118675=11, srv2063678358=63, srv436299316=72, srv1629137915=41, srv1962315526=57, srv1260002150=16, srv1100318467=6, srv1976911852=58, srv645612092=81, srv1945350098=56, srv89266318=92, srv1261585312=17, srv999017102=99, srv297199555=69, srv1887745879=54, srv1804606025=49, srv1627650727=40, srv978136603=95, srv1047012532=5, srv544818308=74, srv1273002670=18, srv1578522914=35, srv570706431=76, srv625192601=80, srv1045514077=4, srv139615934=28, srv1708209537=44, srv1162451994=9, srv1593023053=37, srv403122589=70, srv1594888723=38, srv1041680745=2, srv1120011215=7, srv1843395196=51} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-24 11:53:21,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-24 11:53:21,501 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-24 11:53:21,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-24 11:53:21,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-24 11:53:21,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-24 11:53:21,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-24 11:53:21,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-24 11:53:21,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-24 11:53:21,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-24 11:53:21,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-24 11:53:21,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-24 11:53:21,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-24 11:53:21,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-24 11:53:21,502 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-24 11:53:21,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-24 11:53:21,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-24 11:53:21,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-24 11:53:21,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-24 11:53:21,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-24 11:53:21,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-24 11:53:21,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-24 11:53:21,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-24 11:53:21,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-24 11:53:21,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-24 11:53:21,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-24 11:53:21,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-24 11:53:21,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-24 11:53:21,503 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-24 11:53:21,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-24 11:53:21,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-24 11:53:21,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-24 11:53:21,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-24 11:53:21,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-24 11:53:21,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-24 11:53:21,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-24 11:53:21,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-24 11:53:21,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-24 11:53:21,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-24 11:53:21,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-24 11:53:21,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-24 11:53:21,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-24 11:53:21,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-24 11:53:21,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-24 11:53:21,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-24 11:53:21,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-24 11:53:21,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-24 11:53:21,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-24 11:53:21,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-24 11:53:21,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-24 11:53:21,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-24 11:53:21,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-24 11:53:21,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-24 11:53:21,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-24 11:53:21,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-24 11:53:21,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-24 11:53:21,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-24 11:53:21,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-24 11:53:21,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-24 11:53:21,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-24 11:53:21,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-24 11:53:21,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-24 11:53:21,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-24 11:53:21,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-24 11:53:21,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-24 11:53:21,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-24 11:53:21,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-24 11:53:21,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-24 11:53:21,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-24 11:53:21,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-24 11:53:21,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-24 11:53:21,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-24 11:53:21,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-24 11:53:21,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-24 11:53:21,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-24 11:53:21,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-24 11:53:21,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-24 11:53:21,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-24 11:53:21,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-24 11:53:21,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-24 11:53:21,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-24 11:53:21,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-24 11:53:21,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-24 11:53:21,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-24 11:53:21,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-24 11:53:21,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-24 11:53:21,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-24 11:53:21,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-24 11:53:21,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-24 11:53:21,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-24 11:53:21,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-24 11:53:21,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-24 11:53:21,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-24 11:53:21,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-24 11:53:21,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-24 11:53:21,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-24 11:53:21,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-24 11:53:21,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-24 11:53:21,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-24 11:53:21,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-24 11:53:21,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-24 11:53:21,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-24 11:53:21,725 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039586338417444106, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05547064217817214, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039586338417444106, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17216666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23933333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 11:53:21,725 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-24 11:53:21,726 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17698456036931515, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039586338417444106, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05547064217817214, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039586338417444106, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17216666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23933333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-24 11:53:22,055 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 516 ms to try 5000 different iterations.  Found a solution that moves 2647 regions; Going from a computed imbalance of 0.17698456036931515 to a new imbalance of 0.013841451359929705. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02170383010960535); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03759313164099101, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2941111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02170383010960535); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042166666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 11:53:22,073 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv625192601:80 , srv1261585312:81 , srv1424432622:81 , srv577489820:81 , srv806014939:81 , srv1162451994:82 , srv1327187128:82 , srv1354223332:82 , srv1495253974:82 , srv1593023053:82 , srv1800392445:82 , srv570706431:82 , srv1124849205:83 , srv1179118675:83 , srv1218084619:83 , srv1629137915:83 , srv218118297:83 , srv246969525:83 , srv297199555:83 , srv61780420:83 , srv645612092:83 , srv1045514077:84 , srv1100318467:84 , srv1289087975:84 , srv1578522914:84 , srv1580558849:84 , srv1627650727:84 , srv1962315526:84 , srv284463086:84 , srv1027537536:85 , srv1041680745:85 , srv1316083718:85 , srv1370971554:85 , srv139615934:85 , srv1569638132:85 , srv1601069010:85 , srv1650568028:85 , srv1804606025:85 , srv1945350098:85 , srv1980024641:85 , srv436299316:85 , srv544818308:85 , srv716601816:85 , srv739332935:85 , srv836322980:85 , srv1026452051:86 , srv1044309165:86 , srv1356468339:86 , srv1361928933:86 , srv1430396028:86 , srv1594888723:86 , srv1648040864:86 , srv1865037287:86 , srv1869094201:86 , srv2063678358:86 , srv241693789:86 , srv262201298:86 , srv58651320:86 , srv658598863:86 , srv826579456:86 , srv836720476:86 , srv981724973:86 , srv1163308610:87 , srv1188045530:87 , srv122738517:87 , srv1450411964:87 , srv1576509943:87 , srv174018332:87 , srv1805250207:87 , srv1843395196:87 , srv405277231:87 , srv668278123:87 , srv688793622:87 , srv922137:87 , srv978136603:87 , srv999017102:87 , srv1047012532:88 , srv1120011215:88 , srv1256894853:88 , srv1301606746:88 , srv1708209537:88 , srv1772812900:88 , srv1887745879:88 , srv2012375239:88 , srv458790993:88 , srv560100299:88 , srv89266318:88 , srv988777985:88 , srv1273002670:89 , srv1390273947:89 , srv1902479205:89 , srv1976911852:89 , srv1993405595:89 , srv2034597569:89 , srv403122589:89 , srv957097289:89 , srv997777786:89 , srv66830231:90 , srv1762554429:91 , srv1260002150:525 }
2024-02-24 11:53:22,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-24 11:53:22,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1980024641=59, srv1044309165=3, srv1580558849=36, srv2034597569=62, srv218118297=64, srv1576509943=34, srv1800392445=48, srv922137=93, srv1256894853=15, srv981724973=96, srv1993405595=60, srv836720476=91, srv1865037287=52, srv284463086=68, srv405277231=71, srv668278123=83, srv957097289=94, srv174018332=45, srv806014939=88, srv1869094201=53, srv1424432622=29, srv1354223332=23, srv1027537536=1, srv1218084619=13, srv1772812900=47, srv997777786=98, srv458790993=73, srv560100299=75, srv988777985=97, srv2012375239=61, srv1650568028=43, srv1124849205=8, srv1601069010=39, srv1450411964=31, srv688793622=85, srv1361928933=25, srv1356468339=24, srv1370971554=26, srv122738517=14, srv1188045530=12, srv1569638132=33, srv836322980=90, srv1163308610=10, srv58651320=78, srv1301606746=20, srv1805250207=50, srv739332935=87, srv577489820=77, srv262201298=67, srv1327187128=22, srv716601816=86, srv1902479205=55, srv1026452051=0, srv658598863=82, srv1430396028=30, srv1648040864=42, srv1316083718=21, srv1495253974=32, srv61780420=79, srv1289087975=19, srv1390273947=27, srv241693789=65, srv1762554429=46, srv66830231=84, srv826579456=89, srv246969525=66, srv1179118675=11, srv2063678358=63, srv436299316=72, srv1629137915=41, srv1962315526=57, srv1260002150=16, srv1100318467=6, srv1976911852=58, srv645612092=81, srv1945350098=56, srv89266318=92, srv1261585312=17, srv999017102=99, srv297199555=69, srv1887745879=54, srv1804606025=49, srv1627650727=40, srv978136603=95, srv1047012532=5, srv544818308=74, srv1273002670=18, srv1578522914=35, srv570706431=76, srv625192601=80, srv1045514077=4, srv139615934=28, srv1708209537=44, srv1162451994=9, srv1593023053=37, srv403122589=70, srv1594888723=38, srv1041680745=2, srv1120011215=7, srv1843395196=51} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-24 11:53:22,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-24 11:53:22,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-24 11:53:22,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-24 11:53:22,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-24 11:53:22,096 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-24 11:53:22,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-24 11:53:22,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-24 11:53:22,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-24 11:53:22,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-24 11:53:22,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-24 11:53:22,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-24 11:53:22,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-24 11:53:22,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-24 11:53:22,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-24 11:53:22,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-24 11:53:22,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-24 11:53:22,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-24 11:53:22,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-24 11:53:22,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-24 11:53:22,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-24 11:53:22,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-24 11:53:22,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-24 11:53:22,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-24 11:53:22,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-24 11:53:22,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-24 11:53:22,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-24 11:53:22,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-24 11:53:22,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-24 11:53:22,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-24 11:53:22,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-24 11:53:22,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-24 11:53:22,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-24 11:53:22,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-24 11:53:22,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-24 11:53:22,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-24 11:53:22,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-24 11:53:22,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-24 11:53:22,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-24 11:53:22,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-24 11:53:22,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-24 11:53:22,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-24 11:53:22,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-24 11:53:22,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-24 11:53:22,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-24 11:53:22,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-24 11:53:22,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-24 11:53:22,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-24 11:53:22,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-24 11:53:22,099 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-24 11:53:22,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-24 11:53:22,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-24 11:53:22,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-24 11:53:22,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-24 11:53:22,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-24 11:53:22,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-24 11:53:22,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-24 11:53:22,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-24 11:53:22,100 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-24 11:53:22,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-24 11:53:22,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-24 11:53:22,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-24 11:53:22,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-24 11:53:22,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-24 11:53:22,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-24 11:53:22,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-24 11:53:22,101 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-24 11:53:22,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-24 11:53:22,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-24 11:53:22,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-24 11:53:22,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-24 11:53:22,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-24 11:53:22,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-24 11:53:22,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-24 11:53:22,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-24 11:53:22,102 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-24 11:53:22,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-24 11:53:22,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-24 11:53:22,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-24 11:53:22,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-24 11:53:22,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-24 11:53:22,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-24 11:53:22,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-24 11:53:22,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-24 11:53:22,103 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-24 11:53:22,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-24 11:53:22,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-24 11:53:22,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-24 11:53:22,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-24 11:53:22,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-24 11:53:22,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-24 11:53:22,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-24 11:53:22,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-24 11:53:22,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-24 11:53:22,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-24 11:53:22,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-24 11:53:22,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-24 11:53:22,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-24 11:53:22,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-24 11:53:22,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-24 11:53:22,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-24 11:53:22,105 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-24 11:53:22,275 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02170383010960535); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03759313164099101, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02170383010960535); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042166666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 11:53:22,275 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-24 11:53:22,276 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01382291666977983, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02170383010960535); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03759313164099101, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02170383010960535); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.010833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042166666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-24 11:53:22,737 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 605 ms to try 5000 different iterations.  Found a solution that moves 943 regions; Going from a computed imbalance of 0.01382291666977983 to a new imbalance of 9.552367430468601E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007349945473685019); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011211666202433622); MoveCostFunction : (multiplier=7.0, imbalance=0.10477777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007349945473685019); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 11:53:22,740 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv458790993:82 , srv560100299:82 , srv139615934:83 , srv1578522914:83 , srv1627650727:83 , srv1762554429:83 , srv1772812900:83 , srv1887745879:83 , srv2012375239:83 , srv1162451994:84 , srv1179118675:84 , srv1450411964:84 , srv1594888723:84 , srv1648040864:84 , srv1980024641:84 , srv1993405595:84 , srv2063678358:84 , srv241693789:84 , srv405277231:84 , srv577489820:84 , srv66830231:84 , srv89266318:84 , srv981724973:84 , srv997777786:84 , srv1026452051:85 , srv1041680745:85 , srv1044309165:85 , srv1045514077:85 , srv1047012532:85 , srv1120011215:85 , srv1163308610:85 , srv1256894853:85 , srv1356468339:85 , srv1576509943:85 , srv1580558849:85 , srv1629137915:85 , srv1650568028:85 , srv1708209537:85 , srv1800392445:85 , srv1804606025:85 , srv1805250207:85 , srv1843395196:85 , srv1865037287:85 , srv1945350098:85 , srv1976911852:85 , srv2034597569:85 , srv218118297:85 , srv403122589:85 , srv436299316:85 , srv570706431:85 , srv58651320:85 , srv625192601:85 , srv658598863:85 , srv688793622:85 , srv836322980:85 , srv1027537536:86 , srv1100318467:86 , srv1124849205:86 , srv122738517:86 , srv1261585312:86 , srv1289087975:86 , srv1301606746:86 , srv1316083718:86 , srv1370971554:86 , srv1390273947:86 , srv1424432622:86 , srv1569638132:86 , srv1593023053:86 , srv1601069010:86 , srv174018332:86 , srv1869094201:86 , srv1902479205:86 , srv284463086:86 , srv297199555:86 , srv544818308:86 , srv645612092:86 , srv668278123:86 , srv716601816:86 , srv739332935:86 , srv826579456:86 , srv922137:86 , srv957097289:86 , srv999017102:86 , srv1188045530:87 , srv1218084619:87 , srv1273002670:87 , srv1354223332:87 , srv1361928933:87 , srv1430396028:87 , srv1495253974:87 , srv1962315526:87 , srv246969525:87 , srv61780420:87 , srv806014939:87 , srv836720476:87 , srv988777985:87 , srv1327187128:88 , srv978136603:88 , srv262201298:89 , srv1260002150:556 }
2024-02-24 11:53:22,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-24 11:53:22,748 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1980024641=59, srv1044309165=3, srv1580558849=36, srv2034597569=62, srv218118297=64, srv1576509943=34, srv1800392445=48, srv922137=93, srv1256894853=15, srv981724973=96, srv1993405595=60, srv836720476=91, srv1865037287=52, srv284463086=68, srv405277231=71, srv668278123=83, srv957097289=94, srv174018332=45, srv806014939=88, srv1869094201=53, srv1424432622=29, srv1354223332=23, srv1027537536=1, srv1218084619=13, srv1772812900=47, srv997777786=98, srv458790993=73, srv560100299=75, srv988777985=97, srv2012375239=61, srv1650568028=43, srv1124849205=8, srv1601069010=39, srv1450411964=31, srv688793622=85, srv1361928933=25, srv1356468339=24, srv1370971554=26, srv122738517=14, srv1188045530=12, srv1569638132=33, srv836322980=90, srv1163308610=10, srv58651320=78, srv1301606746=20, srv1805250207=50, srv739332935=87, srv577489820=77, srv262201298=67, srv1327187128=22, srv716601816=86, srv1902479205=55, srv1026452051=0, srv658598863=82, srv1430396028=30, srv1648040864=42, srv1316083718=21, srv1495253974=32, srv61780420=79, srv1289087975=19, srv1390273947=27, srv241693789=65, srv1762554429=46, srv66830231=84, srv826579456=89, srv246969525=66, srv1179118675=11, srv2063678358=63, srv436299316=72, srv1629137915=41, srv1962315526=57, srv1260002150=16, srv1100318467=6, srv1976911852=58, srv645612092=81, srv1945350098=56, srv89266318=92, srv1261585312=17, srv999017102=99, srv297199555=69, srv1887745879=54, srv1804606025=49, srv1627650727=40, srv978136603=95, srv1047012532=5, srv544818308=74, srv1273002670=18, srv1578522914=35, srv570706431=76, srv625192601=80, srv1045514077=4, srv139615934=28, srv1708209537=44, srv1162451994=9, srv1593023053=37, srv403122589=70, srv1594888723=38, srv1041680745=2, srv1120011215=7, srv1843395196=51} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-24 11:53:22,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-24 11:53:22,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-24 11:53:22,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-24 11:53:22,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-24 11:53:22,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-24 11:53:22,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-24 11:53:22,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-24 11:53:22,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-24 11:53:22,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-24 11:53:22,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-24 11:53:22,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-24 11:53:22,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-24 11:53:22,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-24 11:53:22,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-24 11:53:22,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-24 11:53:22,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-24 11:53:22,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-24 11:53:22,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-24 11:53:22,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-24 11:53:22,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-24 11:53:22,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-24 11:53:22,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-24 11:53:22,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-24 11:53:22,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-24 11:53:22,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-24 11:53:22,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-24 11:53:22,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-24 11:53:22,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-24 11:53:22,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-24 11:53:22,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-24 11:53:22,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-24 11:53:22,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-24 11:53:22,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-24 11:53:22,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-24 11:53:22,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-24 11:53:22,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-24 11:53:22,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-24 11:53:22,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-24 11:53:22,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-24 11:53:22,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-24 11:53:22,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-24 11:53:22,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-24 11:53:22,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-24 11:53:22,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-24 11:53:22,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-24 11:53:22,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-24 11:53:22,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-24 11:53:22,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-24 11:53:22,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-24 11:53:22,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-24 11:53:22,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-24 11:53:22,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-24 11:53:22,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-24 11:53:22,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-24 11:53:22,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-24 11:53:22,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-24 11:53:22,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-24 11:53:22,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-24 11:53:22,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-24 11:53:22,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-24 11:53:22,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-24 11:53:22,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-24 11:53:22,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-24 11:53:22,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-24 11:53:22,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-24 11:53:22,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-24 11:53:22,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-24 11:53:22,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-24 11:53:22,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-24 11:53:22,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-24 11:53:22,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-24 11:53:22,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-24 11:53:22,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-24 11:53:22,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-24 11:53:22,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-24 11:53:22,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-24 11:53:22,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-24 11:53:22,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-24 11:53:22,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-24 11:53:22,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-24 11:53:22,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-24 11:53:22,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-24 11:53:22,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-24 11:53:22,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-24 11:53:22,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-24 11:53:22,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-24 11:53:22,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-24 11:53:22,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-24 11:53:22,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-24 11:53:22,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-24 11:53:22,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-24 11:53:22,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-24 11:53:22,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-24 11:53:22,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-24 11:53:22,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-24 11:53:22,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-24 11:53:22,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-24 11:53:22,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-24 11:53:22,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-24 11:53:22,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-24 11:53:22,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-24 11:53:22,900 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007349945473685019); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011211666202433622); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007349945473685019); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 11:53:22,900 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-24 11:53:22,900 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.486337159175315E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007349945473685019); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011211666202433622); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007349945473685019); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-24 11:53:23,210 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 423 ms to try 5000 different iterations.  Found a solution that moves 325 regions; Going from a computed imbalance of 9.486337159175315E-4 to a new imbalance of 3.0466403189791808E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023688968483956715); MoveCostFunction : (multiplier=7.0, imbalance=0.03611111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 11:53:23,211 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv668278123:82 , srv1047012532:83 , srv1289087975:83 , srv1390273947:83 , srv1800392445:83 , srv262201298:83 , srv978136603:83 , srv1026452051:84 , srv1027537536:84 , srv1045514077:84 , srv1124849205:84 , srv1188045530:84 , srv1301606746:84 , srv1316083718:84 , srv1354223332:84 , srv1356468339:84 , srv1424432622:84 , srv1430396028:84 , srv1495253974:84 , srv1593023053:84 , srv1708209537:84 , srv1865037287:84 , srv1902479205:84 , srv1962315526:84 , srv1976911852:84 , srv2063678358:84 , srv241693789:84 , srv297199555:84 , srv403122589:84 , srv436299316:84 , srv577489820:84 , srv58651320:84 , srv658598863:84 , srv688793622:84 , srv716601816:84 , srv836720476:84 , srv922137:84 , srv957097289:84 , srv988777985:84 , srv999017102:84 , srv1041680745:85 , srv1044309165:85 , srv1100318467:85 , srv1120011215:85 , srv1162451994:85 , srv1163308610:85 , srv1179118675:85 , srv1218084619:85 , srv122738517:85 , srv1256894853:85 , srv1273002670:85 , srv1327187128:85 , srv1361928933:85 , srv1370971554:85 , srv1450411964:85 , srv1569638132:85 , srv1578522914:85 , srv1580558849:85 , srv1601069010:85 , srv1627650727:85 , srv1629137915:85 , srv174018332:85 , srv1762554429:85 , srv1772812900:85 , srv1805250207:85 , srv1843395196:85 , srv1945350098:85 , srv1980024641:85 , srv2012375239:85 , srv2034597569:85 , srv246969525:85 , srv284463086:85 , srv405277231:85 , srv458790993:85 , srv544818308:85 , srv560100299:85 , srv570706431:85 , srv625192601:85 , srv66830231:85 , srv739332935:85 , srv806014939:85 , srv826579456:85 , srv836322980:85 , srv981724973:85 , srv997777786:85 , srv1261585312:86 , srv139615934:86 , srv1576509943:86 , srv1594888723:86 , srv1648040864:86 , srv1650568028:86 , srv1804606025:86 , srv1869094201:86 , srv1887745879:86 , srv1993405595:86 , srv218118297:86 , srv61780420:86 , srv645612092:86 , srv89266318:86 , srv1260002150:619 }
2024-02-24 11:53:23,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-24 11:53:23,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1980024641=59, srv1044309165=3, srv1580558849=36, srv2034597569=62, srv218118297=64, srv1576509943=34, srv1800392445=48, srv922137=93, srv1256894853=15, srv981724973=96, srv1993405595=60, srv836720476=91, srv1865037287=52, srv284463086=68, srv405277231=71, srv668278123=83, srv957097289=94, srv174018332=45, srv806014939=88, srv1869094201=53, srv1424432622=29, srv1354223332=23, srv1027537536=1, srv1218084619=13, srv1772812900=47, srv997777786=98, srv458790993=73, srv560100299=75, srv988777985=97, srv2012375239=61, srv1650568028=43, srv1124849205=8, srv1601069010=39, srv1450411964=31, srv688793622=85, srv1361928933=25, srv1356468339=24, srv1370971554=26, srv122738517=14, srv1188045530=12, srv1569638132=33, srv836322980=90, srv1163308610=10, srv58651320=78, srv1301606746=20, srv1805250207=50, srv739332935=87, srv577489820=77, srv262201298=67, srv1327187128=22, srv716601816=86, srv1902479205=55, srv1026452051=0, srv658598863=82, srv1430396028=30, srv1648040864=42, srv1316083718=21, srv1495253974=32, srv61780420=79, srv1289087975=19, srv1390273947=27, srv241693789=65, srv1762554429=46, srv66830231=84, srv826579456=89, srv246969525=66, srv1179118675=11, srv2063678358=63, srv436299316=72, srv1629137915=41, srv1962315526=57, srv1260002150=16, srv1100318467=6, srv1976911852=58, srv645612092=81, srv1945350098=56, srv89266318=92, srv1261585312=17, srv999017102=99, srv297199555=69, srv1887745879=54, srv1804606025=49, srv1627650727=40, srv978136603=95, srv1047012532=5, srv544818308=74, srv1273002670=18, srv1578522914=35, srv570706431=76, srv625192601=80, srv1045514077=4, srv139615934=28, srv1708209537=44, srv1162451994=9, srv1593023053=37, srv403122589=70, srv1594888723=38, srv1041680745=2, srv1120011215=7, srv1843395196=51} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-24 11:53:23,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-24 11:53:23,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-24 11:53:23,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-24 11:53:23,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-24 11:53:23,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-24 11:53:23,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-24 11:53:23,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-24 11:53:23,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-24 11:53:23,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-24 11:53:23,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-24 11:53:23,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-24 11:53:23,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-24 11:53:23,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-24 11:53:23,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-24 11:53:23,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-24 11:53:23,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-24 11:53:23,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-24 11:53:23,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-24 11:53:23,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-24 11:53:23,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-24 11:53:23,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-24 11:53:23,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-24 11:53:23,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-24 11:53:23,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-24 11:53:23,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-24 11:53:23,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-24 11:53:23,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-24 11:53:23,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-24 11:53:23,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-24 11:53:23,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-24 11:53:23,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-24 11:53:23,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-24 11:53:23,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-24 11:53:23,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-24 11:53:23,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-24 11:53:23,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-24 11:53:23,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-24 11:53:23,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-24 11:53:23,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-24 11:53:23,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-24 11:53:23,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-24 11:53:23,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-24 11:53:23,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-24 11:53:23,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-24 11:53:23,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-24 11:53:23,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-24 11:53:23,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-24 11:53:23,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-24 11:53:23,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-24 11:53:23,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-24 11:53:23,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-24 11:53:23,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-24 11:53:23,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-24 11:53:23,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-24 11:53:23,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-24 11:53:23,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-24 11:53:23,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-24 11:53:23,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-24 11:53:23,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-24 11:53:23,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-24 11:53:23,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-24 11:53:23,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-24 11:53:23,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-24 11:53:23,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-24 11:53:23,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-24 11:53:23,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-24 11:53:23,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-24 11:53:23,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-24 11:53:23,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-24 11:53:23,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-24 11:53:23,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-24 11:53:23,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-24 11:53:23,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-24 11:53:23,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-24 11:53:23,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-24 11:53:23,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-24 11:53:23,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-24 11:53:23,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-24 11:53:23,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-24 11:53:23,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-24 11:53:23,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-24 11:53:23,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-24 11:53:23,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-24 11:53:23,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-24 11:53:23,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-24 11:53:23,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-24 11:53:23,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-24 11:53:23,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-24 11:53:23,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-24 11:53:23,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-24 11:53:23,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-24 11:53:23,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-24 11:53:23,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-24 11:53:23,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-24 11:53:23,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-24 11:53:23,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-24 11:53:23,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-24 11:53:23,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-24 11:53:23,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-24 11:53:23,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-24 11:53:23,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-24 11:53:23,347 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023688968483956715); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 11:53:23,347 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-24 11:53:23,347 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.819070455030948E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0023688968483956715); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-24 11:53:23,585 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 346 ms to try 5000 different iterations.  Found a solution that moves 26 regions; Going from a computed imbalance of 2.819070455030948E-5 to a new imbalance of 9.23017400075114E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0028888888888888888); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 11:53:23,586 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1044309165:83 , srv1218084619:83 , srv1361928933:83 , srv1450411964:83 , srv1650568028:83 , srv1804606025:83 , srv1843395196:83 , srv1869094201:83 , srv458790993:83 , srv61780420:83 , srv739332935:83 , srv1026452051:84 , srv1041680745:84 , srv1047012532:84 , srv1100318467:84 , srv1120011215:84 , srv1124849205:84 , srv1162451994:84 , srv1163308610:84 , srv1179118675:84 , srv1188045530:84 , srv122738517:84 , srv1256894853:84 , srv1261585312:84 , srv1273002670:84 , srv1289087975:84 , srv1301606746:84 , srv1316083718:84 , srv1327187128:84 , srv1354223332:84 , srv1370971554:84 , srv1390273947:84 , srv139615934:84 , srv1424432622:84 , srv1430396028:84 , srv1495253974:84 , srv1569638132:84 , srv1576509943:84 , srv1578522914:84 , srv1580558849:84 , srv1593023053:84 , srv1594888723:84 , srv1601069010:84 , srv1629137915:84 , srv1648040864:84 , srv174018332:84 , srv1762554429:84 , srv1772812900:84 , srv1805250207:84 , srv1887745879:84 , srv1902479205:84 , srv1962315526:84 , srv1976911852:84 , srv1980024641:84 , srv1993405595:84 , srv2012375239:84 , srv2034597569:84 , srv2063678358:84 , srv218118297:84 , srv246969525:84 , srv262201298:84 , srv297199555:84 , srv403122589:84 , srv405277231:84 , srv436299316:84 , srv544818308:84 , srv560100299:84 , srv570706431:84 , srv577489820:84 , srv58651320:84 , srv625192601:84 , srv645612092:84 , srv658598863:84 , srv668278123:84 , srv66830231:84 , srv688793622:84 , srv716601816:84 , srv806014939:84 , srv826579456:84 , srv836322980:84 , srv836720476:84 , srv89266318:84 , srv922137:84 , srv978136603:84 , srv981724973:84 , srv988777985:84 , srv997777786:84 , srv999017102:84 , srv1027537536:85 , srv1045514077:85 , srv1356468339:85 , srv1627650727:85 , srv1708209537:85 , srv1800392445:85 , srv1865037287:85 , srv1945350098:85 , srv241693789:85 , srv284463086:85 , srv957097289:85 , srv1260002150:684 }
2024-02-24 11:53:23,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-24 11:53:23,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1980024641=59, srv1044309165=3, srv1580558849=36, srv2034597569=62, srv218118297=64, srv1576509943=34, srv1800392445=48, srv922137=93, srv1256894853=15, srv981724973=96, srv1993405595=60, srv836720476=91, srv1865037287=52, srv284463086=68, srv405277231=71, srv668278123=83, srv957097289=94, srv174018332=45, srv806014939=88, srv1869094201=53, srv1424432622=29, srv1354223332=23, srv1027537536=1, srv1218084619=13, srv1772812900=47, srv997777786=98, srv458790993=73, srv560100299=75, srv988777985=97, srv2012375239=61, srv1650568028=43, srv1124849205=8, srv1601069010=39, srv1450411964=31, srv688793622=85, srv1361928933=25, srv1356468339=24, srv1370971554=26, srv122738517=14, srv1188045530=12, srv1569638132=33, srv836322980=90, srv1163308610=10, srv58651320=78, srv1301606746=20, srv1805250207=50, srv739332935=87, srv577489820=77, srv262201298=67, srv1327187128=22, srv716601816=86, srv1902479205=55, srv1026452051=0, srv658598863=82, srv1430396028=30, srv1648040864=42, srv1316083718=21, srv1495253974=32, srv61780420=79, srv1289087975=19, srv1390273947=27, srv241693789=65, srv1762554429=46, srv66830231=84, srv826579456=89, srv246969525=66, srv1179118675=11, srv2063678358=63, srv436299316=72, srv1629137915=41, srv1962315526=57, srv1260002150=16, srv1100318467=6, srv1976911852=58, srv645612092=81, srv1945350098=56, srv89266318=92, srv1261585312=17, srv999017102=99, srv297199555=69, srv1887745879=54, srv1804606025=49, srv1627650727=40, srv978136603=95, srv1047012532=5, srv544818308=74, srv1273002670=18, srv1578522914=35, srv570706431=76, srv625192601=80, srv1045514077=4, srv139615934=28, srv1708209537=44, srv1162451994=9, srv1593023053=37, srv403122589=70, srv1594888723=38, srv1041680745=2, srv1120011215=7, srv1843395196=51} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-24 11:53:23,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-24 11:53:23,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-24 11:53:23,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-24 11:53:23,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-24 11:53:23,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-24 11:53:23,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-24 11:53:23,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-24 11:53:23,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-24 11:53:23,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-24 11:53:23,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-24 11:53:23,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-24 11:53:23,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-24 11:53:23,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-24 11:53:23,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-24 11:53:23,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-24 11:53:23,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-24 11:53:23,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-24 11:53:23,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-24 11:53:23,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-24 11:53:23,601 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-24 11:53:23,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-24 11:53:23,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-24 11:53:23,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-24 11:53:23,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-24 11:53:23,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-24 11:53:23,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-24 11:53:23,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-24 11:53:23,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-24 11:53:23,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-24 11:53:23,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-24 11:53:23,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-24 11:53:23,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-24 11:53:23,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-24 11:53:23,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-24 11:53:23,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-24 11:53:23,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-24 11:53:23,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-24 11:53:23,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-24 11:53:23,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-24 11:53:23,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-24 11:53:23,602 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-24 11:53:23,603 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-24 11:53:23,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-24 11:53:23,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-24 11:53:23,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-24 11:53:23,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-24 11:53:23,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-24 11:53:23,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-24 11:53:23,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-24 11:53:23,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-24 11:53:23,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-24 11:53:23,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-24 11:53:23,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-24 11:53:23,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-24 11:53:23,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-24 11:53:23,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-24 11:53:23,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-24 11:53:23,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-24 11:53:23,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-24 11:53:23,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-24 11:53:23,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-24 11:53:23,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-24 11:53:23,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-24 11:53:23,604 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-24 11:53:23,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-24 11:53:23,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-24 11:53:23,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-24 11:53:23,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-24 11:53:23,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-24 11:53:23,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-24 11:53:23,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-24 11:53:23,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-24 11:53:23,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-24 11:53:23,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-24 11:53:23,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-24 11:53:23,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-24 11:53:23,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-24 11:53:23,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-24 11:53:23,605 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-24 11:53:23,717 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=9.048118109592553E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 11:53:23,717 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1044309165:83 , srv1218084619:83 , srv1361928933:83 , srv1450411964:83 , srv1650568028:83 , srv1804606025:83 , srv1843395196:83 , srv1869094201:83 , srv458790993:83 , srv61780420:83 , srv739332935:83 , srv1026452051:84 , srv1041680745:84 , srv1047012532:84 , srv1100318467:84 , srv1120011215:84 , srv1124849205:84 , srv1162451994:84 , srv1163308610:84 , srv1179118675:84 , srv1188045530:84 , srv122738517:84 , srv1256894853:84 , srv1261585312:84 , srv1273002670:84 , srv1289087975:84 , srv1301606746:84 , srv1316083718:84 , srv1327187128:84 , srv1354223332:84 , srv1370971554:84 , srv1390273947:84 , srv139615934:84 , srv1424432622:84 , srv1430396028:84 , srv1495253974:84 , srv1569638132:84 , srv1576509943:84 , srv1578522914:84 , srv1580558849:84 , srv1593023053:84 , srv1594888723:84 , srv1601069010:84 , srv1629137915:84 , srv1648040864:84 , srv174018332:84 , srv1762554429:84 , srv1772812900:84 , srv1805250207:84 , srv1887745879:84 , srv1902479205:84 , srv1962315526:84 , srv1976911852:84 , srv1980024641:84 , srv1993405595:84 , srv2012375239:84 , srv2034597569:84 , srv2063678358:84 , srv218118297:84 , srv246969525:84 , srv262201298:84 , srv297199555:84 , srv403122589:84 , srv405277231:84 , srv436299316:84 , srv544818308:84 , srv560100299:84 , srv570706431:84 , srv577489820:84 , srv58651320:84 , srv625192601:84 , srv645612092:84 , srv658598863:84 , srv668278123:84 , srv66830231:84 , srv688793622:84 , srv716601816:84 , srv806014939:84 , srv826579456:84 , srv836322980:84 , srv836720476:84 , srv89266318:84 , srv922137:84 , srv978136603:84 , srv981724973:84 , srv988777985:84 , srv997777786:84 , srv999017102:84 , srv1027537536:85 , srv1045514077:85 , srv1356468339:85 , srv1627650727:85 , srv1708209537:85 , srv1800392445:85 , srv1865037287:85 , srv1945350098:85 , srv241693789:85 , srv284463086:85 , srv957097289:85 , srv1260002150:684 }
2024-02-24 11:53:23,751 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=272 (was 272), ProcessCount=170 (was 170), AvailableMemoryMB=21317 (was 22077)
2024-02-24 11:53:23,760 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=272, ProcessCount=170, AvailableMemoryMB=21316
2024-02-24 11:53:23,761 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-02-24 11:53:23,761 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-02-24 11:53:23,774 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1136646993:84 , srv1272580857:84 , srv1292323588:84 , srv1312124187:84 , srv13897348:84 , srv1390089636:84 , srv1540062680:84 , srv15746238:84 , srv1583611283:84 , srv1586178334:84 , srv160224974:84 , srv1641464282:84 , srv1729733090:84 , srv1832847485:84 , srv1852538826:84 , srv1867668920:84 , srv1893968574:84 , srv1899794469:84 , srv1931483136:84 , srv195372201:84 , srv2033185259:84 , srv2075848295:84 , srv396015329:84 , srv47235208:84 , srv4874303:84 , srv58092148:84 , srv692148924:84 , srv693199417:84 , srv839462853:84 , srv1397428616:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-02-24 11:53:23,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-24 11:53:23,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1390089636=5, srv1931483136=19, srv1272580857=1, srv692148924=27, srv13897348=4, srv1852538826=15, srv2075848295=22, srv1867668920=16, srv1899794469=18, srv1136646993=0, srv1583611283=9, srv15746238=8, srv47235208=24, srv1312124187=3, srv839462853=29, srv195372201=20, srv1832847485=14, srv1729733090=13, srv396015329=23, srv160224974=11, srv1397428616=6, srv1540062680=7, srv1641464282=12, srv693199417=28, srv2033185259=21, srv4874303=25, srv1893968574=17, srv1586178334=10, srv1292323588=2, srv58092148=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-24 11:53:23,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-24 11:53:23,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-24 11:53:23,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-24 11:53:23,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-24 11:53:23,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-24 11:53:23,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-24 11:53:23,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-24 11:53:23,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-24 11:53:23,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-24 11:53:23,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-24 11:53:23,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-24 11:53:23,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-24 11:53:23,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-24 11:53:23,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-24 11:53:23,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-24 11:53:23,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-24 11:53:23,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-24 11:53:23,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-24 11:53:23,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-24 11:53:23,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-24 11:53:23,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-24 11:53:23,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-24 11:53:23,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-24 11:53:23,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-24 11:53:23,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-24 11:53:23,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-24 11:53:23,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-24 11:53:23,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-24 11:53:23,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-24 11:53:23,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-24 11:53:23,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-02-24 11:53:23,803 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.061002277924815, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 11:53:23,803 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912385259450114, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.061002277924815, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-02-24 11:55:23,790 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5534682 different iterations.  Found a solution that moves 1852 regions; Going from a computed imbalance of 0.9912385259450114 to a new imbalance of 4.322660390073086E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6859259259259259, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 11:55:23,795 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1136646993:90 , srv1272580857:90 , srv1292323588:90 , srv1312124187:90 , srv13897348:90 , srv1390089636:90 , srv1397428616:90 , srv1540062680:90 , srv15746238:90 , srv1583611283:90 , srv1586178334:90 , srv160224974:90 , srv1641464282:90 , srv1729733090:90 , srv1832847485:90 , srv1852538826:90 , srv1867668920:90 , srv1893968574:90 , srv1899794469:90 , srv1931483136:90 , srv195372201:90 , srv2033185259:90 , srv2075848295:90 , srv396015329:90 , srv47235208:90 , srv4874303:90 , srv58092148:90 , srv692148924:90 , srv693199417:90 , srv839462853:90 }
2024-02-24 11:55:23,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-24 11:55:23,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1390089636=5, srv1931483136=19, srv1272580857=1, srv692148924=27, srv13897348=4, srv1852538826=15, srv2075848295=22, srv1867668920=16, srv1899794469=18, srv1136646993=0, srv1583611283=9, srv15746238=8, srv47235208=24, srv1312124187=3, srv839462853=29, srv195372201=20, srv1832847485=14, srv1729733090=13, srv396015329=23, srv160224974=11, srv1397428616=6, srv1540062680=7, srv1641464282=12, srv693199417=28, srv2033185259=21, srv4874303=25, srv1893968574=17, srv1586178334=10, srv1292323588=2, srv58092148=26} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-24 11:55:23,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-24 11:55:23,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-24 11:55:23,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-24 11:55:23,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-24 11:55:23,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-24 11:55:23,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-24 11:55:23,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-24 11:55:23,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-24 11:55:23,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-24 11:55:23,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-24 11:55:23,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-24 11:55:23,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-24 11:55:23,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-24 11:55:23,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-24 11:55:23,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-24 11:55:23,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-24 11:55:23,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-24 11:55:23,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-24 11:55:23,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-24 11:55:23,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-24 11:55:23,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-24 11:55:23,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-24 11:55:23,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-24 11:55:23,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-24 11:55:23,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-24 11:55:23,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-24 11:55:23,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-24 11:55:23,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-24 11:55:23,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-24 11:55:23,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-24 11:55:23,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-02-24 11:55:23,813 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 11:55:23,814 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1136646993:90 , srv1272580857:90 , srv1292323588:90 , srv1312124187:90 , srv13897348:90 , srv1390089636:90 , srv1397428616:90 , srv1540062680:90 , srv15746238:90 , srv1583611283:90 , srv1586178334:90 , srv160224974:90 , srv1641464282:90 , srv1729733090:90 , srv1832847485:90 , srv1852538826:90 , srv1867668920:90 , srv1893968574:90 , srv1899794469:90 , srv1931483136:90 , srv195372201:90 , srv2033185259:90 , srv2075848295:90 , srv396015329:90 , srv47235208:90 , srv4874303:90 , srv58092148:90 , srv692148924:90 , srv693199417:90 , srv839462853:90 }
2024-02-24 11:55:23,830 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=155 (was 272), ProcessCount=168 (was 170), AvailableMemoryMB=20914 (was 21316)