2023-11-29 15:53:12,136 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-11-29 15:53:12,462 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-11-29 15:53:12,581 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-11-29 15:53:12,646 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-29 15:53:12,653 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-29 15:53:12,671 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=325, ProcessCount=170, AvailableMemoryMB=10816
2023-11-29 15:53:12,674 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-29 15:53:12,674 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-29 15:53:12,816 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1070400008:84 , srv1077293222:84 , srv1084068628:84 , srv1114495576:84 , srv1121345248:84 , srv1128866052:84 , srv113149834:84 , srv116400824:84 , srv1187193027:84 , srv1197503734:84 , srv1200165584:84 , srv1206066317:84 , srv1240146125:84 , srv1245663221:84 , srv1281173099:84 , srv1285644252:84 , srv1295970798:84 , srv1306744804:84 , srv1345671076:84 , srv1364370020:84 , srv1379555956:84 , srv1384708986:84 , srv1399670936:84 , srv1427430842:84 , srv143200097:84 , srv1457463199:84 , srv1471594953:84 , srv1492677975:84 , srv1519465546:84 , srv1538454514:84 , srv1543445926:84 , srv1558882145:84 , srv1575625658:84 , srv1576916163:84 , srv1588062876:84 , srv1593657581:84 , srv1596705709:84 , srv1607757416:84 , srv16153201:84 , srv1621391302:84 , srv1634505763:84 , srv1641544270:84 , srv1666332740:84 , srv169257615:84 , srv1712483812:84 , srv1744322752:84 , srv1755602423:84 , srv1756140080:84 , srv176750470:84 , srv177054619:84 , srv1805597236:84 , srv1824185091:84 , srv1890397259:84 , srv1891142918:84 , srv1901605862:84 , srv1948416997:84 , srv2015347856:84 , srv2035521100:84 , srv2050119703:84 , srv2054202242:84 , srv2057029953:84 , srv2098096679:84 , srv2116457667:84 , srv2132837107:84 , srv2136478934:84 , srv2146244152:84 , srv228884175:84 , srv2293292:84 , srv261254882:84 , srv264102407:84 , srv265138612:84 , srv278124700:84 , srv2785657:84 , srv281204466:84 , srv341236484:84 , srv357325541:84 , srv363199903:84 , srv432255145:84 , srv483624317:84 , srv494903069:84 , srv504276187:84 , srv522407156:84 , srv529902374:84 , srv542128264:84 , srv542309061:84 , srv544786463:84 , srv553550968:84 , srv587100072:84 , srv599964092:84 , srv62640163:84 , srv651194034:84 , srv65984940:84 , srv730679157:84 , srv759555349:84 , srv768213908:84 , srv77859350:84 , srv812010664:84 , srv99066495:84 , srv994882643:84 , srv192030012:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-11-29 15:53:12,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-29 15:53:12,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1295970798=16, srv228884175=67, srv1427430842=23, srv176750470=48, srv169257615=43, srv261254882=69, srv1206066317=11, srv599964092=89, srv1641544270=41, srv1666332740=42, srv494903069=80, srv1901605862=54, srv542128264=84, srv768213908=95, srv1345671076=18, srv1281173099=14, srv99066495=98, srv357325541=76, srv2054202242=60, srv542309061=85, srv2057029953=61, srv1588062876=34, srv730679157=93, srv113149834=6, srv1285644252=15, srv1805597236=50, srv1756140080=47, srv363199903=77, srv65984940=92, srv1364370020=19, srv1621391302=39, srv1824185091=51, srv2098096679=62, srv1593657581=35, srv1471594953=26, srv1948416997=56, srv2050119703=59, srv1576916163=33, srv1634505763=40, srv1399670936=22, srv1084068628=2, srv62640163=90, srv16153201=38, srv1306744804=17, srv2132837107=64, srv1128866052=5, srv529902374=83, srv1077293222=1, srv1200165584=10, srv2785657=73, srv812010664=97, srv2116457667=63, srv1197503734=9, srv1121345248=4, srv1379555956=20, srv1492677975=27, srv759555349=94, srv432255145=78, srv1712483812=44, srv192030012=55, srv2293292=68, srv994882643=99, srv1890397259=52, srv2136478934=65, srv1607757416=37, srv278124700=72, srv651194034=91, srv264102407=70, srv1070400008=0, srv1187193027=8, srv522407156=82, srv1457463199=25, srv2035521100=58, srv544786463=86, srv265138612=71, srv1596705709=36, srv1744322752=45, srv1245663221=13, srv1384708986=21, srv1575625658=32, srv1114495576=3, srv1519465546=28, srv77859350=96, srv1891142918=53, srv1543445926=30, srv1558882145=31, srv2015347856=57, srv281204466=74, srv587100072=88, srv177054619=49, srv1755602423=46, srv143200097=24, srv1538454514=29, srv504276187=81, srv483624317=79, srv116400824=7, srv2146244152=66, srv341236484=75, srv1240146125=12, srv553550968=87} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-29 15:53:12,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-29 15:53:12,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-29 15:53:12,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-29 15:53:12,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-29 15:53:12,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-29 15:53:12,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-29 15:53:12,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-29 15:53:12,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-29 15:53:12,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-29 15:53:12,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-29 15:53:12,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-29 15:53:12,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-29 15:53:12,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-29 15:53:12,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-29 15:53:12,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-29 15:53:12,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-29 15:53:12,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-29 15:53:12,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-29 15:53:12,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-29 15:53:12,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-29 15:53:12,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-29 15:53:12,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-29 15:53:12,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-29 15:53:12,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-29 15:53:12,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-29 15:53:12,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-29 15:53:12,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-29 15:53:12,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-29 15:53:12,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-29 15:53:12,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-29 15:53:12,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-29 15:53:12,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-29 15:53:12,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-29 15:53:12,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-29 15:53:12,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-29 15:53:12,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-29 15:53:12,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-29 15:53:12,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-29 15:53:12,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-29 15:53:12,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-29 15:53:12,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-29 15:53:12,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-29 15:53:12,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-29 15:53:12,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-29 15:53:12,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-29 15:53:12,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-29 15:53:12,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-29 15:53:12,920 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-29 15:53:12,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-29 15:53:12,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-29 15:53:12,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-29 15:53:12,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-29 15:53:12,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-29 15:53:12,921 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-29 15:53:12,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-29 15:53:12,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-29 15:53:12,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-29 15:53:12,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-29 15:53:12,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-29 15:53:12,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-29 15:53:12,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-29 15:53:12,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-29 15:53:12,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-29 15:53:12,922 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-29 15:53:12,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-29 15:53:12,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-29 15:53:12,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-29 15:53:12,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-29 15:53:12,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-29 15:53:12,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-29 15:53:12,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-29 15:53:12,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-29 15:53:12,923 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-29 15:53:12,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-29 15:53:12,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-29 15:53:12,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-29 15:53:12,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-29 15:53:12,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-29 15:53:12,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-29 15:53:12,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-29 15:53:12,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-29 15:53:12,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-29 15:53:12,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-29 15:53:12,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-29 15:53:12,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-29 15:53:12,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-29 15:53:12,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-29 15:53:12,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-29 15:53:12,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-29 15:53:12,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-29 15:53:12,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-29 15:53:12,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-29 15:53:12,926 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-29 15:53:12,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-29 15:53:12,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-29 15:53:12,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-29 15:53:12,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-29 15:53:12,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-29 15:53:12,927 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-29 15:53:12,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-29 15:53:12,928 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-29 15:53:13,180 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 15:53:13,181 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-29 15:53:13,181 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-29 15:53:13,655 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 699 ms to try 5000 different iterations.  Found a solution that moves 4691 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17877549868018586. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037346518434880696, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053434247961572874, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5212222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.037346518434880696, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17425, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23825, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 15:53:13,697 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1200165584:82 , srv1607757416:82 , srv65984940:82 , srv1345671076:83 , srv1824185091:83 , srv1891142918:83 , srv2050119703:83 , srv278124700:83 , srv544786463:83 , srv1206066317:84 , srv1240146125:84 , srv1364370020:84 , srv1543445926:84 , srv177054619:84 , srv264102407:84 , srv357325541:84 , srv542309061:84 , srv1306744804:85 , srv143200097:85 , srv1492677975:85 , srv1538454514:85 , srv1558882145:85 , srv16153201:85 , srv1755602423:85 , srv1756140080:85 , srv1890397259:85 , srv1948416997:85 , srv2057029953:85 , srv2116457667:85 , srv261254882:85 , srv363199903:85 , srv494903069:85 , srv1084068628:86 , srv116400824:86 , srv1295970798:86 , srv1427430842:86 , srv1471594953:86 , srv1901605862:86 , srv2098096679:86 , srv2785657:86 , srv341236484:86 , srv483624317:86 , srv504276187:86 , srv529902374:86 , srv99066495:86 , srv1114495576:87 , srv1121345248:87 , srv1281173099:87 , srv1285644252:87 , srv1399670936:87 , srv1457463199:87 , srv1519465546:87 , srv1666332740:87 , srv1712483812:87 , srv1805597236:87 , srv2015347856:87 , srv2054202242:87 , srv2146244152:87 , srv432255145:87 , srv522407156:87 , srv553550968:87 , srv62640163:87 , srv730679157:87 , srv759555349:87 , srv768213908:87 , srv994882643:87 , srv1077293222:88 , srv1187193027:88 , srv1197503734:88 , srv1575625658:88 , srv1593657581:88 , srv1596705709:88 , srv1634505763:88 , srv1641544270:88 , srv1744322752:88 , srv176750470:88 , srv2035521100:88 , srv542128264:88 , srv77859350:88 , srv1245663221:89 , srv1379555956:89 , srv169257615:89 , srv2132837107:89 , srv228884175:89 , srv2293292:89 , srv265138612:89 , srv281204466:89 , srv587100072:89 , srv599964092:89 , srv1070400008:90 , srv1128866052:90 , srv113149834:90 , srv1384708986:90 , srv1576916163:90 , srv2136478934:91 , srv651194034:91 , srv1588062876:92 , srv1621391302:92 , srv812010664:92 , srv192030012:422 }
2023-11-29 15:53:13,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-29 15:53:13,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1295970798=16, srv228884175=67, srv1427430842=23, srv176750470=48, srv169257615=43, srv261254882=69, srv1206066317=11, srv599964092=89, srv1641544270=41, srv1666332740=42, srv494903069=80, srv1901605862=54, srv542128264=84, srv768213908=95, srv1345671076=18, srv1281173099=14, srv99066495=98, srv357325541=76, srv2054202242=60, srv542309061=85, srv2057029953=61, srv1588062876=34, srv730679157=93, srv113149834=6, srv1285644252=15, srv1805597236=50, srv1756140080=47, srv363199903=77, srv65984940=92, srv1364370020=19, srv1621391302=39, srv1824185091=51, srv2098096679=62, srv1593657581=35, srv1471594953=26, srv1948416997=56, srv2050119703=59, srv1576916163=33, srv1634505763=40, srv1399670936=22, srv1084068628=2, srv62640163=90, srv16153201=38, srv1306744804=17, srv2132837107=64, srv1128866052=5, srv529902374=83, srv1077293222=1, srv1200165584=10, srv2785657=73, srv812010664=97, srv2116457667=63, srv1197503734=9, srv1121345248=4, srv1379555956=20, srv1492677975=27, srv759555349=94, srv432255145=78, srv1712483812=44, srv192030012=55, srv2293292=68, srv994882643=99, srv1890397259=52, srv2136478934=65, srv1607757416=37, srv278124700=72, srv651194034=91, srv264102407=70, srv1070400008=0, srv1187193027=8, srv522407156=82, srv1457463199=25, srv2035521100=58, srv544786463=86, srv265138612=71, srv1596705709=36, srv1744322752=45, srv1245663221=13, srv1384708986=21, srv1575625658=32, srv1114495576=3, srv1519465546=28, srv77859350=96, srv1891142918=53, srv1543445926=30, srv1558882145=31, srv2015347856=57, srv281204466=74, srv587100072=88, srv177054619=49, srv1755602423=46, srv143200097=24, srv1538454514=29, srv504276187=81, srv483624317=79, srv116400824=7, srv2146244152=66, srv341236484=75, srv1240146125=12, srv553550968=87} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-29 15:53:13,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-29 15:53:13,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-29 15:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-29 15:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-29 15:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-29 15:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-29 15:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-29 15:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-29 15:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-29 15:53:13,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-29 15:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-29 15:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-29 15:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-29 15:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-29 15:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-29 15:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-29 15:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-29 15:53:13,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-29 15:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-29 15:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-29 15:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-29 15:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-29 15:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-29 15:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-29 15:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-29 15:53:13,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-29 15:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-29 15:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-29 15:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-29 15:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-29 15:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-29 15:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-29 15:53:13,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-29 15:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-29 15:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-29 15:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-29 15:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-29 15:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-29 15:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-29 15:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-29 15:53:13,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-29 15:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-29 15:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-29 15:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-29 15:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-29 15:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-29 15:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-29 15:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-29 15:53:13,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-29 15:53:13,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-29 15:53:13,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-29 15:53:13,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-29 15:53:13,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-29 15:53:13,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-29 15:53:13,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-29 15:53:13,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-29 15:53:13,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-29 15:53:13,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-29 15:53:13,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-29 15:53:13,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-29 15:53:13,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-29 15:53:13,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-29 15:53:13,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-29 15:53:13,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-29 15:53:13,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-29 15:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-29 15:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-29 15:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-29 15:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-29 15:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-29 15:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-29 15:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-29 15:53:13,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-29 15:53:13,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-29 15:53:13,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-29 15:53:13,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-29 15:53:13,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-29 15:53:13,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-29 15:53:13,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-29 15:53:13,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-29 15:53:13,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-29 15:53:13,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-29 15:53:13,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-29 15:53:13,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-29 15:53:13,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-29 15:53:13,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-29 15:53:13,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-29 15:53:13,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-29 15:53:13,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-29 15:53:13,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-29 15:53:13,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-29 15:53:13,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-29 15:53:13,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-29 15:53:13,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-29 15:53:13,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-29 15:53:13,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-29 15:53:13,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-29 15:53:13,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-29 15:53:13,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-29 15:53:13,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-29 15:53:13,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-29 15:53:13,913 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037346518434880696, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053434247961572874, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.037346518434880696, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17425, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23825, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 15:53:13,914 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-29 15:53:13,914 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17874265159613106, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037346518434880696, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053434247961572874, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.037346518434880696, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17425, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23825, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-29 15:53:14,279 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 506 ms to try 5000 different iterations.  Found a solution that moves 2625 regions; Going from a computed imbalance of 0.17874265159613106 to a new imbalance of 0.016207946764171678. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.017914736190902298); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02907477105686266, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2916666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.017914736190902298); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 15:53:14,292 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1588062876:79 , srv1128866052:81 , srv651194034:81 , srv1070400008:82 , srv1384708986:82 , srv1596705709:82 , srv1621391302:82 , srv169257615:82 , srv2035521100:82 , srv281204466:82 , srv812010664:82 , srv1187193027:83 , srv1197503734:83 , srv1457463199:83 , srv1805597236:83 , srv2136478934:83 , srv2146244152:83 , srv228884175:83 , srv265138612:83 , srv587100072:83 , srv759555349:83 , srv1084068628:84 , srv113149834:84 , srv1245663221:84 , srv1281173099:84 , srv1285644252:84 , srv1399670936:84 , srv1634505763:84 , srv1744322752:84 , srv2132837107:84 , srv2293292:84 , srv432255145:84 , srv542128264:84 , srv730679157:84 , srv116400824:85 , srv1427430842:85 , srv1575625658:85 , srv1593657581:85 , srv1641544270:85 , srv1712483812:85 , srv599964092:85 , srv62640163:85 , srv768213908:85 , srv77859350:85 , srv99066495:85 , srv1077293222:86 , srv1295970798:86 , srv1379555956:86 , srv143200097:86 , srv1492677975:86 , srv1576916163:86 , srv16153201:86 , srv1666332740:86 , srv176750470:86 , srv1901605862:86 , srv2098096679:86 , srv2116457667:86 , srv483624317:86 , srv494903069:86 , srv553550968:86 , srv994882643:86 , srv1114495576:87 , srv1121345248:87 , srv1519465546:87 , srv1543445926:87 , srv1756140080:87 , srv2054202242:87 , srv261254882:87 , srv2785657:87 , srv522407156:87 , srv544786463:87 , srv1240146125:88 , srv1306744804:88 , srv1345671076:88 , srv1538454514:88 , srv1755602423:88 , srv1948416997:88 , srv2057029953:88 , srv341236484:88 , srv357325541:88 , srv363199903:88 , srv504276187:88 , srv1206066317:89 , srv1364370020:89 , srv1471594953:89 , srv1558882145:89 , srv1824185091:89 , srv1890397259:89 , srv2050119703:89 , srv264102407:89 , srv529902374:89 , srv542309061:89 , srv1200165584:90 , srv1607757416:90 , srv177054619:90 , srv1891142918:90 , srv2015347856:90 , srv65984940:90 , srv278124700:91 , srv192030012:511 }
2023-11-29 15:53:14,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-29 15:53:14,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1295970798=16, srv228884175=67, srv1427430842=23, srv176750470=48, srv169257615=43, srv261254882=69, srv1206066317=11, srv599964092=89, srv1641544270=41, srv1666332740=42, srv494903069=80, srv1901605862=54, srv542128264=84, srv768213908=95, srv1345671076=18, srv1281173099=14, srv99066495=98, srv357325541=76, srv2054202242=60, srv542309061=85, srv2057029953=61, srv1588062876=34, srv730679157=93, srv113149834=6, srv1285644252=15, srv1805597236=50, srv1756140080=47, srv363199903=77, srv65984940=92, srv1364370020=19, srv1621391302=39, srv1824185091=51, srv2098096679=62, srv1593657581=35, srv1471594953=26, srv1948416997=56, srv2050119703=59, srv1576916163=33, srv1634505763=40, srv1399670936=22, srv1084068628=2, srv62640163=90, srv16153201=38, srv1306744804=17, srv2132837107=64, srv1128866052=5, srv529902374=83, srv1077293222=1, srv1200165584=10, srv2785657=73, srv812010664=97, srv2116457667=63, srv1197503734=9, srv1121345248=4, srv1379555956=20, srv1492677975=27, srv759555349=94, srv432255145=78, srv1712483812=44, srv192030012=55, srv2293292=68, srv994882643=99, srv1890397259=52, srv2136478934=65, srv1607757416=37, srv278124700=72, srv651194034=91, srv264102407=70, srv1070400008=0, srv1187193027=8, srv522407156=82, srv1457463199=25, srv2035521100=58, srv544786463=86, srv265138612=71, srv1596705709=36, srv1744322752=45, srv1245663221=13, srv1384708986=21, srv1575625658=32, srv1114495576=3, srv1519465546=28, srv77859350=96, srv1891142918=53, srv1543445926=30, srv1558882145=31, srv2015347856=57, srv281204466=74, srv587100072=88, srv177054619=49, srv1755602423=46, srv143200097=24, srv1538454514=29, srv504276187=81, srv483624317=79, srv116400824=7, srv2146244152=66, srv341236484=75, srv1240146125=12, srv553550968=87} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-29 15:53:14,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-29 15:53:14,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-29 15:53:14,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-29 15:53:14,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-29 15:53:14,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-29 15:53:14,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-29 15:53:14,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-29 15:53:14,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-29 15:53:14,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-29 15:53:14,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-29 15:53:14,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-29 15:53:14,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-29 15:53:14,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-29 15:53:14,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-29 15:53:14,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-29 15:53:14,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-29 15:53:14,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-29 15:53:14,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-29 15:53:14,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-29 15:53:14,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-29 15:53:14,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-29 15:53:14,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-29 15:53:14,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-29 15:53:14,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-29 15:53:14,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-29 15:53:14,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-29 15:53:14,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-29 15:53:14,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-29 15:53:14,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-29 15:53:14,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-29 15:53:14,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-29 15:53:14,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-29 15:53:14,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-29 15:53:14,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-29 15:53:14,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-29 15:53:14,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-29 15:53:14,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-29 15:53:14,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-29 15:53:14,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-29 15:53:14,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-29 15:53:14,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-29 15:53:14,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-29 15:53:14,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-29 15:53:14,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-29 15:53:14,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-29 15:53:14,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-29 15:53:14,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-29 15:53:14,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-29 15:53:14,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-29 15:53:14,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-29 15:53:14,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-29 15:53:14,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-29 15:53:14,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-29 15:53:14,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-29 15:53:14,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-29 15:53:14,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-29 15:53:14,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-29 15:53:14,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-29 15:53:14,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-29 15:53:14,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-29 15:53:14,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-29 15:53:14,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-29 15:53:14,345 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-29 15:53:14,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-29 15:53:14,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-29 15:53:14,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-29 15:53:14,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-29 15:53:14,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-29 15:53:14,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-29 15:53:14,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-29 15:53:14,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-29 15:53:14,346 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-29 15:53:14,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-29 15:53:14,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-29 15:53:14,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-29 15:53:14,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-29 15:53:14,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-29 15:53:14,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-29 15:53:14,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-29 15:53:14,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-29 15:53:14,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-29 15:53:14,347 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-29 15:53:14,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-29 15:53:14,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-29 15:53:14,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-29 15:53:14,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-29 15:53:14,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-29 15:53:14,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-29 15:53:14,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-29 15:53:14,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-29 15:53:14,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-29 15:53:14,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-29 15:53:14,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-29 15:53:14,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-29 15:53:14,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-29 15:53:14,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-29 15:53:14,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-29 15:53:14,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-29 15:53:14,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-29 15:53:14,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-29 15:53:14,350 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-29 15:53:14,476 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.017914736190902298); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02907477105686266, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.017914736190902298); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 15:53:14,477 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-29 15:53:14,477 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016189566121314322, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.017914736190902298); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02907477105686266, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.017914736190902298); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044083333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-29 15:53:14,760 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 393 ms to try 5000 different iterations.  Found a solution that moves 1028 regions; Going from a computed imbalance of 0.016189566121314322 to a new imbalance of 6.620464492191111E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005213965342504877); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006564879518897745); MoveCostFunction : (multiplier=7.0, imbalance=0.11422222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005213965342504877); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0033333333333333335); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 15:53:14,762 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2015347856:80 , srv1576916163:81 , srv1379555956:82 , srv529902374:82 , srv1077293222:83 , srv1121345248:83 , srv1519465546:83 , srv1621391302:83 , srv1890397259:83 , srv2054202242:83 , srv341236484:83 , srv812010664:83 , srv113149834:84 , srv1206066317:84 , srv1240146125:84 , srv1306744804:84 , srv1471594953:84 , srv1558882145:84 , srv1641544270:84 , srv176750470:84 , srv177054619:84 , srv2132837107:84 , srv264102407:84 , srv278124700:84 , srv2785657:84 , srv363199903:84 , srv504276187:84 , srv522407156:84 , srv553550968:84 , srv599964092:84 , srv994882643:84 , srv1070400008:85 , srv1114495576:85 , srv1295970798:85 , srv1364370020:85 , srv1384708986:85 , srv143200097:85 , srv1538454514:85 , srv1575625658:85 , srv1593657581:85 , srv1607757416:85 , srv1634505763:85 , srv1666332740:85 , srv1712483812:85 , srv1755602423:85 , srv1824185091:85 , srv1891142918:85 , srv2057029953:85 , srv2098096679:85 , srv2136478934:85 , srv2293292:85 , srv265138612:85 , srv542128264:85 , srv587100072:85 , srv651194034:85 , srv65984940:85 , srv768213908:85 , srv77859350:85 , srv1128866052:86 , srv1200165584:86 , srv1245663221:86 , srv1281173099:86 , srv1285644252:86 , srv1345671076:86 , srv1427430842:86 , srv1492677975:86 , srv1588062876:86 , srv16153201:86 , srv1744322752:86 , srv1901605862:86 , srv1948416997:86 , srv2050119703:86 , srv2116457667:86 , srv228884175:86 , srv261254882:86 , srv357325541:86 , srv483624317:86 , srv494903069:86 , srv542309061:86 , srv62640163:86 , srv1084068628:87 , srv116400824:87 , srv1187193027:87 , srv1197503734:87 , srv1399670936:87 , srv1543445926:87 , srv1596705709:87 , srv169257615:87 , srv1756140080:87 , srv2146244152:87 , srv281204466:87 , srv432255145:87 , srv730679157:87 , srv759555349:87 , srv99066495:87 , srv1457463199:88 , srv1805597236:88 , srv2035521100:88 , srv544786463:88 , srv192030012:571 }
2023-11-29 15:53:14,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-29 15:53:14,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1295970798=16, srv228884175=67, srv1427430842=23, srv176750470=48, srv169257615=43, srv261254882=69, srv1206066317=11, srv599964092=89, srv1641544270=41, srv1666332740=42, srv494903069=80, srv1901605862=54, srv542128264=84, srv768213908=95, srv1345671076=18, srv1281173099=14, srv99066495=98, srv357325541=76, srv2054202242=60, srv542309061=85, srv2057029953=61, srv1588062876=34, srv730679157=93, srv113149834=6, srv1285644252=15, srv1805597236=50, srv1756140080=47, srv363199903=77, srv65984940=92, srv1364370020=19, srv1621391302=39, srv1824185091=51, srv2098096679=62, srv1593657581=35, srv1471594953=26, srv1948416997=56, srv2050119703=59, srv1576916163=33, srv1634505763=40, srv1399670936=22, srv1084068628=2, srv62640163=90, srv16153201=38, srv1306744804=17, srv2132837107=64, srv1128866052=5, srv529902374=83, srv1077293222=1, srv1200165584=10, srv2785657=73, srv812010664=97, srv2116457667=63, srv1197503734=9, srv1121345248=4, srv1379555956=20, srv1492677975=27, srv759555349=94, srv432255145=78, srv1712483812=44, srv192030012=55, srv2293292=68, srv994882643=99, srv1890397259=52, srv2136478934=65, srv1607757416=37, srv278124700=72, srv651194034=91, srv264102407=70, srv1070400008=0, srv1187193027=8, srv522407156=82, srv1457463199=25, srv2035521100=58, srv544786463=86, srv265138612=71, srv1596705709=36, srv1744322752=45, srv1245663221=13, srv1384708986=21, srv1575625658=32, srv1114495576=3, srv1519465546=28, srv77859350=96, srv1891142918=53, srv1543445926=30, srv1558882145=31, srv2015347856=57, srv281204466=74, srv587100072=88, srv177054619=49, srv1755602423=46, srv143200097=24, srv1538454514=29, srv504276187=81, srv483624317=79, srv116400824=7, srv2146244152=66, srv341236484=75, srv1240146125=12, srv553550968=87} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-29 15:53:14,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-29 15:53:14,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-29 15:53:14,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-29 15:53:14,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-29 15:53:14,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-29 15:53:14,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-29 15:53:14,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-29 15:53:14,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-29 15:53:14,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-29 15:53:14,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-29 15:53:14,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-29 15:53:14,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-29 15:53:14,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-29 15:53:14,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-29 15:53:14,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-29 15:53:14,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-29 15:53:14,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-29 15:53:14,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-29 15:53:14,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-29 15:53:14,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-29 15:53:14,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-29 15:53:14,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-29 15:53:14,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-29 15:53:14,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-29 15:53:14,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-29 15:53:14,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-29 15:53:14,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-29 15:53:14,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-29 15:53:14,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-29 15:53:14,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-29 15:53:14,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-29 15:53:14,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-29 15:53:14,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-29 15:53:14,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-29 15:53:14,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-29 15:53:14,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-29 15:53:14,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-29 15:53:14,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-29 15:53:14,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-29 15:53:14,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-29 15:53:14,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-29 15:53:14,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-29 15:53:14,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-29 15:53:14,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-29 15:53:14,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-29 15:53:14,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-29 15:53:14,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-29 15:53:14,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-29 15:53:14,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-29 15:53:14,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-29 15:53:14,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-29 15:53:14,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-29 15:53:14,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-29 15:53:14,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-29 15:53:14,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-29 15:53:14,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-29 15:53:14,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-29 15:53:14,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-29 15:53:14,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-29 15:53:14,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-29 15:53:14,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-29 15:53:14,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-29 15:53:14,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-29 15:53:14,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-29 15:53:14,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-29 15:53:14,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-29 15:53:14,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-29 15:53:14,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-29 15:53:14,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-29 15:53:14,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-29 15:53:14,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-29 15:53:14,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-29 15:53:14,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-29 15:53:14,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-29 15:53:14,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-29 15:53:14,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-29 15:53:14,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-29 15:53:14,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-29 15:53:14,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-29 15:53:14,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-29 15:53:14,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-29 15:53:14,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-29 15:53:14,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-29 15:53:14,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-29 15:53:14,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-29 15:53:14,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-29 15:53:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-29 15:53:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-29 15:53:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-29 15:53:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-29 15:53:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-29 15:53:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-29 15:53:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-29 15:53:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-29 15:53:14,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-29 15:53:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-29 15:53:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-29 15:53:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-29 15:53:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-29 15:53:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-29 15:53:14,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-29 15:53:14,910 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005213965342504877); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006564879518897745); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005213965342504877); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0033333333333333335); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 15:53:14,910 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-29 15:53:14,910 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.548482393686872E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005213965342504877); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006564879518897745); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005213965342504877); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0033333333333333335); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-29 15:53:15,351 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 539 ms to try 5000 different iterations.  Found a solution that moves 188 regions; Going from a computed imbalance of 6.548482393686872E-4 to a new imbalance of 1.0527124751826907E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.020888888888888887); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 15:53:15,352 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1114495576:83 , srv1245663221:83 , srv1471594953:83 , srv1756140080:83 , srv1948416997:83 , srv2136478934:83 , srv542309061:83 , srv1128866052:84 , srv113149834:84 , srv116400824:84 , srv1187193027:84 , srv1197503734:84 , srv1364370020:84 , srv1427430842:84 , srv1457463199:84 , srv1538454514:84 , srv1543445926:84 , srv1558882145:84 , srv1575625658:84 , srv1593657581:84 , srv1596705709:84 , srv1621391302:84 , srv1666332740:84 , srv169257615:84 , srv1744322752:84 , srv1755602423:84 , srv176750470:84 , srv1805597236:84 , srv1891142918:84 , srv1901605862:84 , srv2035521100:84 , srv2050119703:84 , srv2057029953:84 , srv228884175:84 , srv2293292:84 , srv261254882:84 , srv278124700:84 , srv281204466:84 , srv357325541:84 , srv432255145:84 , srv483624317:84 , srv504276187:84 , srv522407156:84 , srv544786463:84 , srv599964092:84 , srv62640163:84 , srv730679157:84 , srv77859350:84 , srv99066495:84 , srv1070400008:85 , srv1084068628:85 , srv1121345248:85 , srv1200165584:85 , srv1206066317:85 , srv1281173099:85 , srv1285644252:85 , srv1295970798:85 , srv1306744804:85 , srv1345671076:85 , srv1379555956:85 , srv1384708986:85 , srv1399670936:85 , srv1492677975:85 , srv1519465546:85 , srv1576916163:85 , srv1588062876:85 , srv1607757416:85 , srv16153201:85 , srv1634505763:85 , srv1641544270:85 , srv1712483812:85 , srv177054619:85 , srv1824185091:85 , srv1890397259:85 , srv2015347856:85 , srv2054202242:85 , srv2098096679:85 , srv2116457667:85 , srv2132837107:85 , srv2146244152:85 , srv264102407:85 , srv265138612:85 , srv2785657:85 , srv341236484:85 , srv363199903:85 , srv494903069:85 , srv529902374:85 , srv542128264:85 , srv553550968:85 , srv587100072:85 , srv651194034:85 , srv65984940:85 , srv759555349:85 , srv768213908:85 , srv812010664:85 , srv994882643:85 , srv1077293222:86 , srv1240146125:86 , srv143200097:86 , srv192030012:638 }
2023-11-29 15:53:15,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-29 15:53:15,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1295970798=16, srv228884175=67, srv1427430842=23, srv176750470=48, srv169257615=43, srv261254882=69, srv1206066317=11, srv599964092=89, srv1641544270=41, srv1666332740=42, srv494903069=80, srv1901605862=54, srv542128264=84, srv768213908=95, srv1345671076=18, srv1281173099=14, srv99066495=98, srv357325541=76, srv2054202242=60, srv542309061=85, srv2057029953=61, srv1588062876=34, srv730679157=93, srv113149834=6, srv1285644252=15, srv1805597236=50, srv1756140080=47, srv363199903=77, srv65984940=92, srv1364370020=19, srv1621391302=39, srv1824185091=51, srv2098096679=62, srv1593657581=35, srv1471594953=26, srv1948416997=56, srv2050119703=59, srv1576916163=33, srv1634505763=40, srv1399670936=22, srv1084068628=2, srv62640163=90, srv16153201=38, srv1306744804=17, srv2132837107=64, srv1128866052=5, srv529902374=83, srv1077293222=1, srv1200165584=10, srv2785657=73, srv812010664=97, srv2116457667=63, srv1197503734=9, srv1121345248=4, srv1379555956=20, srv1492677975=27, srv759555349=94, srv432255145=78, srv1712483812=44, srv192030012=55, srv2293292=68, srv994882643=99, srv1890397259=52, srv2136478934=65, srv1607757416=37, srv278124700=72, srv651194034=91, srv264102407=70, srv1070400008=0, srv1187193027=8, srv522407156=82, srv1457463199=25, srv2035521100=58, srv544786463=86, srv265138612=71, srv1596705709=36, srv1744322752=45, srv1245663221=13, srv1384708986=21, srv1575625658=32, srv1114495576=3, srv1519465546=28, srv77859350=96, srv1891142918=53, srv1543445926=30, srv1558882145=31, srv2015347856=57, srv281204466=74, srv587100072=88, srv177054619=49, srv1755602423=46, srv143200097=24, srv1538454514=29, srv504276187=81, srv483624317=79, srv116400824=7, srv2146244152=66, srv341236484=75, srv1240146125=12, srv553550968=87} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-29 15:53:15,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-29 15:53:15,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-29 15:53:15,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-29 15:53:15,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-29 15:53:15,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-29 15:53:15,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-29 15:53:15,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-29 15:53:15,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-29 15:53:15,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-29 15:53:15,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-29 15:53:15,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-29 15:53:15,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-29 15:53:15,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-29 15:53:15,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-29 15:53:15,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-29 15:53:15,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-29 15:53:15,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-29 15:53:15,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-29 15:53:15,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-29 15:53:15,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-29 15:53:15,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-29 15:53:15,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-29 15:53:15,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-29 15:53:15,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-29 15:53:15,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-29 15:53:15,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-29 15:53:15,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-29 15:53:15,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-29 15:53:15,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-29 15:53:15,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-29 15:53:15,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-29 15:53:15,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-29 15:53:15,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-29 15:53:15,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-29 15:53:15,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-29 15:53:15,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-29 15:53:15,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-29 15:53:15,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-29 15:53:15,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-29 15:53:15,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-29 15:53:15,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-29 15:53:15,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-29 15:53:15,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-29 15:53:15,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-29 15:53:15,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-29 15:53:15,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-29 15:53:15,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-29 15:53:15,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-29 15:53:15,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-29 15:53:15,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-29 15:53:15,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-29 15:53:15,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-29 15:53:15,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-29 15:53:15,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-29 15:53:15,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-29 15:53:15,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-29 15:53:15,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-29 15:53:15,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-29 15:53:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-29 15:53:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-29 15:53:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-29 15:53:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-29 15:53:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-29 15:53:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-29 15:53:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-29 15:53:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-29 15:53:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-29 15:53:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-29 15:53:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-29 15:53:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-29 15:53:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-29 15:53:15,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-29 15:53:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-29 15:53:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-29 15:53:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-29 15:53:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-29 15:53:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-29 15:53:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-29 15:53:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-29 15:53:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-29 15:53:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-29 15:53:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-29 15:53:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-29 15:53:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-29 15:53:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-29 15:53:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-29 15:53:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-29 15:53:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-29 15:53:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-29 15:53:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-29 15:53:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-29 15:53:15,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-29 15:53:15,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-29 15:53:15,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-29 15:53:15,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-29 15:53:15,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-29 15:53:15,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-29 15:53:15,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-29 15:53:15,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-29 15:53:15,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-29 15:53:15,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-29 15:53:15,482 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=9.210720615757133E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 15:53:15,483 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1114495576:83 , srv1245663221:83 , srv1471594953:83 , srv1756140080:83 , srv1948416997:83 , srv2136478934:83 , srv542309061:83 , srv1128866052:84 , srv113149834:84 , srv116400824:84 , srv1187193027:84 , srv1197503734:84 , srv1364370020:84 , srv1427430842:84 , srv1457463199:84 , srv1538454514:84 , srv1543445926:84 , srv1558882145:84 , srv1575625658:84 , srv1593657581:84 , srv1596705709:84 , srv1621391302:84 , srv1666332740:84 , srv169257615:84 , srv1744322752:84 , srv1755602423:84 , srv176750470:84 , srv1805597236:84 , srv1891142918:84 , srv1901605862:84 , srv2035521100:84 , srv2050119703:84 , srv2057029953:84 , srv228884175:84 , srv2293292:84 , srv261254882:84 , srv278124700:84 , srv281204466:84 , srv357325541:84 , srv432255145:84 , srv483624317:84 , srv504276187:84 , srv522407156:84 , srv544786463:84 , srv599964092:84 , srv62640163:84 , srv730679157:84 , srv77859350:84 , srv99066495:84 , srv1070400008:85 , srv1084068628:85 , srv1121345248:85 , srv1200165584:85 , srv1206066317:85 , srv1281173099:85 , srv1285644252:85 , srv1295970798:85 , srv1306744804:85 , srv1345671076:85 , srv1379555956:85 , srv1384708986:85 , srv1399670936:85 , srv1492677975:85 , srv1519465546:85 , srv1576916163:85 , srv1588062876:85 , srv1607757416:85 , srv16153201:85 , srv1634505763:85 , srv1641544270:85 , srv1712483812:85 , srv177054619:85 , srv1824185091:85 , srv1890397259:85 , srv2015347856:85 , srv2054202242:85 , srv2098096679:85 , srv2116457667:85 , srv2132837107:85 , srv2146244152:85 , srv264102407:85 , srv265138612:85 , srv2785657:85 , srv341236484:85 , srv363199903:85 , srv494903069:85 , srv529902374:85 , srv542128264:85 , srv553550968:85 , srv587100072:85 , srv651194034:85 , srv65984940:85 , srv759555349:85 , srv768213908:85 , srv812010664:85 , srv994882643:85 , srv1077293222:86 , srv1240146125:86 , srv143200097:86 , srv192030012:638 }
2023-11-29 15:53:15,523 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=325 (was 325), ProcessCount=170 (was 170), AvailableMemoryMB=9947 (was 10816)
2023-11-29 15:53:15,531 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=325, ProcessCount=170, AvailableMemoryMB=9947
2023-11-29 15:53:15,532 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-29 15:53:15,532 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-11-29 15:53:15,545 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1107328147:84 , srv1123444204:84 , srv1292962953:84 , srv1358681547:84 , srv1489935005:84 , srv154293522:84 , srv1555884382:84 , srv1608204660:84 , srv1614623760:84 , srv168578887:84 , srv1698450746:84 , srv1765411449:84 , srv1843503421:84 , srv1882243610:84 , srv1940788319:84 , srv2040429590:84 , srv2051838521:84 , srv2142386107:84 , srv214623005:84 , srv374079593:84 , srv457437449:84 , srv565459687:84 , srv622717658:84 , srv719431615:84 , srv7990579:84 , srv829060765:84 , srv849082307:84 , srv88890337:84 , srv893027086:84 , srv1940166920:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-11-29 15:53:15,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-29 15:53:15,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1940788319=15, srv1940166920=14, srv1123444204=1, srv1292962953=2, srv7990579=25, srv1489935005=4, srv1765411449=11, srv1555884382=6, srv1698450746=10, srv829060765=26, srv1608204660=7, srv719431615=24, srv1614623760=8, srv1107328147=0, srv214623005=19, srv2142386107=18, srv374079593=20, srv1843503421=12, srv154293522=5, srv2040429590=16, srv565459687=22, srv893027086=29, srv1882243610=13, srv1358681547=3, srv849082307=27, srv168578887=9, srv622717658=23, srv88890337=28, srv2051838521=17, srv457437449=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-29 15:53:15,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-29 15:53:15,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-29 15:53:15,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-29 15:53:15,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-29 15:53:15,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-29 15:53:15,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-29 15:53:15,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-29 15:53:15,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-29 15:53:15,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-29 15:53:15,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-29 15:53:15,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-29 15:53:15,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-29 15:53:15,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-29 15:53:15,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-29 15:53:15,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-29 15:53:15,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-29 15:53:15,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-29 15:53:15,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-29 15:53:15,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-29 15:53:15,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-29 15:53:15,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-29 15:53:15,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-29 15:53:15,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-29 15:53:15,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-29 15:53:15,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-29 15:53:15,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-29 15:53:15,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-29 15:53:15,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-29 15:53:15,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-29 15:53:15,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-29 15:53:15,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-29 15:53:15,591 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0695657184120564, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 15:53:15,591 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912412242571468, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0695657184120564, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-11-29 15:55:15,569 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5643870 different iterations.  Found a solution that moves 1862 regions; Going from a computed imbalance of 0.9912412242571468 to a new imbalance of 4.346000888939571E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6896296296296296, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 15:55:15,575 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1107328147:90 , srv1123444204:90 , srv1292962953:90 , srv1358681547:90 , srv1489935005:90 , srv154293522:90 , srv1555884382:90 , srv1608204660:90 , srv1614623760:90 , srv168578887:90 , srv1698450746:90 , srv1765411449:90 , srv1843503421:90 , srv1882243610:90 , srv1940166920:90 , srv1940788319:90 , srv2040429590:90 , srv2051838521:90 , srv2142386107:90 , srv214623005:90 , srv374079593:90 , srv457437449:90 , srv565459687:90 , srv622717658:90 , srv719431615:90 , srv7990579:90 , srv829060765:90 , srv849082307:90 , srv88890337:90 , srv893027086:90 }
2023-11-29 15:55:15,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-29 15:55:15,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1940788319=15, srv1940166920=14, srv1123444204=1, srv1292962953=2, srv7990579=25, srv1489935005=4, srv1765411449=11, srv1555884382=6, srv1698450746=10, srv829060765=26, srv1608204660=7, srv719431615=24, srv1614623760=8, srv1107328147=0, srv214623005=19, srv2142386107=18, srv374079593=20, srv1843503421=12, srv154293522=5, srv2040429590=16, srv565459687=22, srv893027086=29, srv1882243610=13, srv1358681547=3, srv849082307=27, srv168578887=9, srv622717658=23, srv88890337=28, srv2051838521=17, srv457437449=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-29 15:55:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-29 15:55:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-29 15:55:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-29 15:55:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-29 15:55:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-29 15:55:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-29 15:55:15,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-29 15:55:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-29 15:55:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-29 15:55:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-29 15:55:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-29 15:55:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-29 15:55:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-29 15:55:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-29 15:55:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-29 15:55:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-29 15:55:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-29 15:55:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-29 15:55:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-29 15:55:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-29 15:55:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-29 15:55:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-29 15:55:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-29 15:55:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-29 15:55:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-29 15:55:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-29 15:55:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-29 15:55:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-29 15:55:15,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-29 15:55:15,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-29 15:55:15,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-29 15:55:15,592 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-29 15:55:15,592 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1107328147:90 , srv1123444204:90 , srv1292962953:90 , srv1358681547:90 , srv1489935005:90 , srv154293522:90 , srv1555884382:90 , srv1608204660:90 , srv1614623760:90 , srv168578887:90 , srv1698450746:90 , srv1765411449:90 , srv1843503421:90 , srv1882243610:90 , srv1940166920:90 , srv1940788319:90 , srv2040429590:90 , srv2051838521:90 , srv2142386107:90 , srv214623005:90 , srv374079593:90 , srv457437449:90 , srv565459687:90 , srv622717658:90 , srv719431615:90 , srv7990579:90 , srv829060765:90 , srv849082307:90 , srv88890337:90 , srv893027086:90 }
2023-11-29 15:55:15,604 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=145 (was 325), ProcessCount=171 (was 170) - ProcessCount LEAK? -, AvailableMemoryMB=9247 (was 9947)