2024-03-07 07:55:58,281 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-03-07 07:55:58,607 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-03-07 07:55:58,729 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-03-07 07:55:58,797 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-07 07:55:58,805 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-07 07:55:58,821 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=190, ProcessCount=168, AvailableMemoryMB=2072
2024-03-07 07:55:58,825 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-07 07:55:58,825 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-07 07:55:58,956 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1011607232:84 , srv1024465243:84 , srv1097422360:84 , srv115004759:84 , srv1159259820:84 , srv1162142536:84 , srv1185984832:84 , srv122095055:84 , srv1221570247:84 , srv1225957302:84 , srv1240033938:84 , srv124533164:84 , srv1283186923:84 , srv1287679772:84 , srv1290773933:84 , srv1321175387:84 , srv1355117153:84 , srv1377722616:84 , srv138178180:84 , srv1400322858:84 , srv147355003:84 , srv1475812271:84 , srv1533074117:84 , srv1537143367:84 , srv1544124103:84 , srv158676659:84 , srv1588807871:84 , srv1600615414:84 , srv1629740695:84 , srv1691499283:84 , srv1695587777:84 , srv1751270757:84 , srv1785602387:84 , srv1801323483:84 , srv1825520115:84 , srv1833794316:84 , srv1853516268:84 , srv1870930230:84 , srv1884071972:84 , srv1885789381:84 , srv1907636599:84 , srv1912837672:84 , srv1933765852:84 , srv1953549157:84 , srv1967210897:84 , srv1967619596:84 , srv197396424:84 , srv1978777344:84 , srv2014227646:84 , srv2029533346:84 , srv2031921526:84 , srv2039907837:84 , srv205451246:84 , srv2055266885:84 , srv2064569061:84 , srv2091889995:84 , srv2118049052:84 , srv2132411364:84 , srv2140446375:84 , srv239967283:84 , srv288609276:84 , srv300019186:84 , srv303934430:84 , srv326544333:84 , srv332990:84 , srv343132610:84 , srv366344324:84 , srv366469408:84 , srv394109862:84 , srv413288559:84 , srv431535873:84 , srv433922279:84 , srv449830008:84 , srv514152907:84 , srv525176650:84 , srv54702485:84 , srv549669212:84 , srv577919360:84 , srv586267026:84 , srv661354489:84 , srv736879647:84 , srv770691415:84 , srv791263481:84 , srv805231099:84 , srv812723856:84 , srv828318618:84 , srv829031170:84 , srv84218784:84 , srv851534272:84 , srv861602959:84 , srv867737394:84 , srv88401195:84 , srv885165423:84 , srv885705143:84 , srv921643377:84 , srv960023767:84 , srv979036438:84 , srv994816805:84 , srv998834441:84 , srv921035954:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-03-07 07:55:58,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 07:55:58,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1537143367=23, srv1912837672=41, srv433922279=71, srv1801323483=33, srv1162142536=5, srv1695587777=30, srv138178180=18, srv770691415=81, srv1287679772=13, srv1240033938=10, srv829031170=86, srv885705143=93, srv1544124103=24, srv885165423=92, srv1825520115=34, srv805231099=83, srv1400322858=19, srv1967619596=45, srv1290773933=14, srv1691499283=29, srv994816805=98, srv1588807871=26, srv1283186923=12, srv1321175387=15, srv2039907837=51, srv828318618=85, srv851534272=88, srv147355003=20, srv2091889995=55, srv586267026=78, srv791263481=82, srv1097422360=2, srv1377722616=17, srv861602959=89, srv1870930230=37, srv736879647=80, srv2118049052=56, srv549669212=76, srv921035954=94, srv1629740695=28, srv300019186=61, srv326544333=63, srv1884071972=38, srv366344324=66, srv514152907=73, srv1221570247=8, srv1159259820=4, srv812723856=84, srv54702485=75, srv979036438=97, srv2014227646=48, srv2140446375=58, srv1024465243=1, srv115004759=3, srv2055266885=53, srv960023767=96, srv332990=64, srv1785602387=32, srv1600615414=27, srv525176650=74, srv2064569061=54, srv303934430=62, srv1853516268=36, srv124533164=11, srv413288559=69, srv205451246=52, srv1833794316=35, srv88401195=91, srv449830008=72, srv366469408=67, srv867737394=90, srv1953549157=43, srv998834441=99, srv2031921526=50, srv1933765852=42, srv84218784=87, srv1885789381=39, srv1355117153=16, srv394109862=68, srv2029533346=49, srv1978777344=47, srv2132411364=57, srv343132610=65, srv1751270757=31, srv239967283=59, srv1907636599=40, srv1225957302=9, srv197396424=46, srv288609276=60, srv1475812271=21, srv1185984832=6, srv122095055=7, srv431535873=70, srv661354489=79, srv1533074117=22, srv577919360=77, srv158676659=25, srv1967210897=44, srv921643377=95, srv1011607232=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 07:55:59,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 07:55:59,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 07:55:59,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 07:55:59,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 07:55:59,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 07:55:59,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 07:55:59,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 07:55:59,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 07:55:59,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 07:55:59,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 07:55:59,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 07:55:59,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 07:55:59,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 07:55:59,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 07:55:59,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 07:55:59,064 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 07:55:59,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 07:55:59,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 07:55:59,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 07:55:59,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 07:55:59,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 07:55:59,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 07:55:59,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 07:55:59,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 07:55:59,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 07:55:59,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 07:55:59,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 07:55:59,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 07:55:59,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 07:55:59,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 07:55:59,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-07 07:55:59,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-07 07:55:59,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-07 07:55:59,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-07 07:55:59,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-07 07:55:59,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-07 07:55:59,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-07 07:55:59,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-07 07:55:59,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-07 07:55:59,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-07 07:55:59,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-07 07:55:59,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-07 07:55:59,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-07 07:55:59,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-07 07:55:59,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-07 07:55:59,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-07 07:55:59,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-07 07:55:59,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-07 07:55:59,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-07 07:55:59,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-07 07:55:59,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-07 07:55:59,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-07 07:55:59,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-07 07:55:59,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-07 07:55:59,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-07 07:55:59,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-07 07:55:59,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-07 07:55:59,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-07 07:55:59,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-07 07:55:59,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-07 07:55:59,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-07 07:55:59,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-07 07:55:59,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-07 07:55:59,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-07 07:55:59,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-07 07:55:59,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-07 07:55:59,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-07 07:55:59,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-07 07:55:59,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-07 07:55:59,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-07 07:55:59,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-07 07:55:59,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-07 07:55:59,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-07 07:55:59,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-07 07:55:59,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-07 07:55:59,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-07 07:55:59,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-07 07:55:59,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-07 07:55:59,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-07 07:55:59,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-07 07:55:59,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-07 07:55:59,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-07 07:55:59,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-07 07:55:59,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-07 07:55:59,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-07 07:55:59,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-07 07:55:59,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-07 07:55:59,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-07 07:55:59,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-07 07:55:59,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-07 07:55:59,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-07 07:55:59,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-07 07:55:59,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-07 07:55:59,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-07 07:55:59,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-07 07:55:59,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-07 07:55:59,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-07 07:55:59,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-07 07:55:59,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-07 07:55:59,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-07 07:55:59,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-07 07:55:59,338 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 07:55:59,338 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-07 07:55:59,338 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-07 07:55:59,696 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 584 ms to try 5000 different iterations.  Found a solution that moves 4716 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17331921962857044. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041112172765131356, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0522318925067555, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.524, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041112172765131356, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16866666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23333333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 07:55:59,735 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1221570247:81 , srv577919360:81 , srv115004759:82 , srv147355003:82 , srv1600615414:82 , srv2029533346:82 , srv366469408:82 , srv770691415:82 , srv88401195:82 , srv239967283:83 , srv288609276:83 , srv84218784:83 , srv851534272:83 , srv1024465243:84 , srv1162142536:84 , srv122095055:84 , srv1377722616:84 , srv1751270757:84 , srv1801323483:84 , srv1967210897:84 , srv1978777344:84 , srv300019186:84 , srv332990:84 , srv433922279:84 , srv661354489:84 , srv812723856:84 , srv885705143:84 , srv979036438:84 , srv1011607232:85 , srv1283186923:85 , srv1475812271:85 , srv1537143367:85 , srv1629740695:85 , srv1691499283:85 , srv1785602387:85 , srv1833794316:85 , srv1885789381:85 , srv1933765852:85 , srv1953549157:85 , srv2039907837:85 , srv2055266885:85 , srv2091889995:85 , srv791263481:85 , srv805231099:85 , srv921643377:85 , srv1240033938:86 , srv124533164:86 , srv1355117153:86 , srv138178180:86 , srv1400322858:86 , srv1533074117:86 , srv158676659:86 , srv1884071972:86 , srv1907636599:86 , srv2031921526:86 , srv326544333:86 , srv394109862:86 , srv514152907:86 , srv828318618:86 , srv829031170:86 , srv994816805:86 , srv1097422360:87 , srv1544124103:87 , srv1695587777:87 , srv1825520115:87 , srv1853516268:87 , srv2118049052:87 , srv303934430:87 , srv449830008:87 , srv549669212:87 , srv736879647:87 , srv861602959:87 , srv960023767:87 , srv1912837672:88 , srv1967619596:88 , srv2064569061:88 , srv2132411364:88 , srv54702485:88 , srv867737394:88 , srv885165423:88 , srv1321175387:89 , srv2014227646:89 , srv343132610:89 , srv525176650:89 , srv998834441:89 , srv1185984832:90 , srv1290773933:90 , srv1870930230:90 , srv1159259820:91 , srv1225957302:91 , srv1287679772:91 , srv586267026:91 , srv1588807871:92 , srv366344324:92 , srv413288559:92 , srv431535873:92 , srv197396424:95 , srv205451246:96 , srv2140446375:98 , srv921035954:455 }
2024-03-07 07:55:59,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 07:55:59,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1537143367=23, srv1912837672=41, srv433922279=71, srv1801323483=33, srv1162142536=5, srv1695587777=30, srv138178180=18, srv770691415=81, srv1287679772=13, srv1240033938=10, srv829031170=86, srv885705143=93, srv1544124103=24, srv885165423=92, srv1825520115=34, srv805231099=83, srv1400322858=19, srv1967619596=45, srv1290773933=14, srv1691499283=29, srv994816805=98, srv1588807871=26, srv1283186923=12, srv1321175387=15, srv2039907837=51, srv828318618=85, srv851534272=88, srv147355003=20, srv2091889995=55, srv586267026=78, srv791263481=82, srv1097422360=2, srv1377722616=17, srv861602959=89, srv1870930230=37, srv736879647=80, srv2118049052=56, srv549669212=76, srv921035954=94, srv1629740695=28, srv300019186=61, srv326544333=63, srv1884071972=38, srv366344324=66, srv514152907=73, srv1221570247=8, srv1159259820=4, srv812723856=84, srv54702485=75, srv979036438=97, srv2014227646=48, srv2140446375=58, srv1024465243=1, srv115004759=3, srv2055266885=53, srv960023767=96, srv332990=64, srv1785602387=32, srv1600615414=27, srv525176650=74, srv2064569061=54, srv303934430=62, srv1853516268=36, srv124533164=11, srv413288559=69, srv205451246=52, srv1833794316=35, srv88401195=91, srv449830008=72, srv366469408=67, srv867737394=90, srv1953549157=43, srv998834441=99, srv2031921526=50, srv1933765852=42, srv84218784=87, srv1885789381=39, srv1355117153=16, srv394109862=68, srv2029533346=49, srv1978777344=47, srv2132411364=57, srv343132610=65, srv1751270757=31, srv239967283=59, srv1907636599=40, srv1225957302=9, srv197396424=46, srv288609276=60, srv1475812271=21, srv1185984832=6, srv122095055=7, srv431535873=70, srv661354489=79, srv1533074117=22, srv577919360=77, srv158676659=25, srv1967210897=44, srv921643377=95, srv1011607232=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 07:55:59,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 07:55:59,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 07:55:59,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 07:55:59,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 07:55:59,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 07:55:59,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 07:55:59,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 07:55:59,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 07:55:59,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 07:55:59,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 07:55:59,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 07:55:59,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 07:55:59,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 07:55:59,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 07:55:59,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 07:55:59,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 07:55:59,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 07:55:59,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 07:55:59,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 07:55:59,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 07:55:59,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 07:55:59,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 07:55:59,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 07:55:59,767 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 07:55:59,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 07:55:59,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 07:55:59,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 07:55:59,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 07:55:59,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 07:55:59,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 07:55:59,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-07 07:55:59,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-07 07:55:59,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-07 07:55:59,768 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-07 07:55:59,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-07 07:55:59,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-07 07:55:59,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-07 07:55:59,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-07 07:55:59,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-07 07:55:59,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-07 07:55:59,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-07 07:55:59,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-07 07:55:59,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-07 07:55:59,769 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-07 07:55:59,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-07 07:55:59,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-07 07:55:59,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-07 07:55:59,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-07 07:55:59,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-07 07:55:59,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-07 07:55:59,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-07 07:55:59,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-07 07:55:59,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-07 07:55:59,770 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-07 07:55:59,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-07 07:55:59,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-07 07:55:59,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-07 07:55:59,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-07 07:55:59,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-07 07:55:59,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-07 07:55:59,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-07 07:55:59,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-07 07:55:59,771 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-07 07:55:59,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-07 07:55:59,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-07 07:55:59,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-07 07:55:59,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-07 07:55:59,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-07 07:55:59,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-07 07:55:59,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-07 07:55:59,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-07 07:55:59,772 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-07 07:55:59,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-07 07:55:59,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-07 07:55:59,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-07 07:55:59,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-07 07:55:59,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-07 07:55:59,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-07 07:55:59,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-07 07:55:59,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-07 07:55:59,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-07 07:55:59,773 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-07 07:55:59,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-07 07:55:59,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-07 07:55:59,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-07 07:55:59,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-07 07:55:59,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-07 07:55:59,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-07 07:55:59,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-07 07:55:59,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-07 07:55:59,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-07 07:55:59,774 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-07 07:55:59,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-07 07:55:59,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-07 07:55:59,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-07 07:55:59,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-07 07:55:59,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-07 07:55:59,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-07 07:55:59,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-07 07:55:59,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-07 07:55:59,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-07 07:55:59,913 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041112172765131356, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0522318925067555, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041112172765131356, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16866666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23333333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 07:55:59,914 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-07 07:55:59,914 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17328619749077415, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041112172765131356, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0522318925067555, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041112172765131356, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16866666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23333333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-07 07:56:00,143 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 341 ms to try 5000 different iterations.  Found a solution that moves 2612 regions; Going from a computed imbalance of 0.17328619749077415 to a new imbalance of 0.014671239193772395. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021257393834088253); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029132616006393625, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2902222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021257393834088253); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04516666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 07:56:00,152 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2140446375:75 , srv205451246:76 , srv197396424:78 , srv1225957302:80 , srv413288559:80 , srv431535873:80 , srv1159259820:81 , srv1290773933:81 , srv1588807871:81 , srv586267026:81 , srv1185984832:82 , srv1912837672:82 , srv1287679772:83 , srv2132411364:83 , srv366344324:83 , srv525176650:83 , srv54702485:83 , srv549669212:83 , srv998834441:83 , srv1097422360:84 , srv1544124103:84 , srv1870930230:84 , srv1967619596:84 , srv2014227646:84 , srv2064569061:84 , srv303934430:84 , srv343132610:84 , srv394109862:84 , srv449830008:84 , srv867737394:84 , srv1240033938:85 , srv1321175387:85 , srv1475812271:85 , srv1884071972:85 , srv2091889995:85 , srv514152907:85 , srv828318618:85 , srv885165423:85 , srv138178180:86 , srv1400322858:86 , srv1691499283:86 , srv1695587777:86 , srv1853516268:86 , srv1907636599:86 , srv2031921526:86 , srv2118049052:86 , srv326544333:86 , srv805231099:86 , srv921643377:86 , srv960023767:86 , srv994816805:86 , srv122095055:87 , srv1283186923:87 , srv1533074117:87 , srv1537143367:87 , srv158676659:87 , srv1629740695:87 , srv1801323483:87 , srv1825520115:87 , srv1833794316:87 , srv1885789381:87 , srv332990:87 , srv736879647:87 , srv791263481:87 , srv861602959:87 , srv979036438:87 , srv115004759:88 , srv124533164:88 , srv1377722616:88 , srv1953549157:88 , srv1967210897:88 , srv2039907837:88 , srv2055266885:88 , srv239967283:88 , srv288609276:88 , srv300019186:88 , srv433922279:88 , srv661354489:88 , srv770691415:88 , srv829031170:88 , srv84218784:88 , srv88401195:88 , srv885705143:88 , srv1011607232:89 , srv1162142536:89 , srv1355117153:89 , srv1751270757:89 , srv1933765852:89 , srv1978777344:89 , srv2029533346:89 , srv366469408:89 , srv577919360:89 , srv812723856:89 , srv851534272:89 , srv1024465243:90 , srv1221570247:90 , srv1600615414:90 , srv1785602387:90 , srv147355003:92 , srv921035954:508 }
2024-03-07 07:56:00,156 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 07:56:00,157 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1537143367=23, srv1912837672=41, srv433922279=71, srv1801323483=33, srv1162142536=5, srv1695587777=30, srv138178180=18, srv770691415=81, srv1287679772=13, srv1240033938=10, srv829031170=86, srv885705143=93, srv1544124103=24, srv885165423=92, srv1825520115=34, srv805231099=83, srv1400322858=19, srv1967619596=45, srv1290773933=14, srv1691499283=29, srv994816805=98, srv1588807871=26, srv1283186923=12, srv1321175387=15, srv2039907837=51, srv828318618=85, srv851534272=88, srv147355003=20, srv2091889995=55, srv586267026=78, srv791263481=82, srv1097422360=2, srv1377722616=17, srv861602959=89, srv1870930230=37, srv736879647=80, srv2118049052=56, srv549669212=76, srv921035954=94, srv1629740695=28, srv300019186=61, srv326544333=63, srv1884071972=38, srv366344324=66, srv514152907=73, srv1221570247=8, srv1159259820=4, srv812723856=84, srv54702485=75, srv979036438=97, srv2014227646=48, srv2140446375=58, srv1024465243=1, srv115004759=3, srv2055266885=53, srv960023767=96, srv332990=64, srv1785602387=32, srv1600615414=27, srv525176650=74, srv2064569061=54, srv303934430=62, srv1853516268=36, srv124533164=11, srv413288559=69, srv205451246=52, srv1833794316=35, srv88401195=91, srv449830008=72, srv366469408=67, srv867737394=90, srv1953549157=43, srv998834441=99, srv2031921526=50, srv1933765852=42, srv84218784=87, srv1885789381=39, srv1355117153=16, srv394109862=68, srv2029533346=49, srv1978777344=47, srv2132411364=57, srv343132610=65, srv1751270757=31, srv239967283=59, srv1907636599=40, srv1225957302=9, srv197396424=46, srv288609276=60, srv1475812271=21, srv1185984832=6, srv122095055=7, srv431535873=70, srv661354489=79, srv1533074117=22, srv577919360=77, srv158676659=25, srv1967210897=44, srv921643377=95, srv1011607232=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 07:56:00,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 07:56:00,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 07:56:00,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 07:56:00,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 07:56:00,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 07:56:00,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 07:56:00,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 07:56:00,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 07:56:00,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 07:56:00,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 07:56:00,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 07:56:00,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 07:56:00,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 07:56:00,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 07:56:00,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 07:56:00,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 07:56:00,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 07:56:00,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 07:56:00,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 07:56:00,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 07:56:00,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 07:56:00,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 07:56:00,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 07:56:00,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 07:56:00,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 07:56:00,176 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 07:56:00,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 07:56:00,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 07:56:00,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 07:56:00,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 07:56:00,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-07 07:56:00,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-07 07:56:00,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-07 07:56:00,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-07 07:56:00,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-07 07:56:00,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-07 07:56:00,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-07 07:56:00,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-07 07:56:00,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-07 07:56:00,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-07 07:56:00,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-07 07:56:00,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-07 07:56:00,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-07 07:56:00,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-07 07:56:00,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-07 07:56:00,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-07 07:56:00,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-07 07:56:00,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-07 07:56:00,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-07 07:56:00,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-07 07:56:00,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-07 07:56:00,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-07 07:56:00,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-07 07:56:00,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-07 07:56:00,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-07 07:56:00,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-07 07:56:00,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-07 07:56:00,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-07 07:56:00,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-07 07:56:00,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-07 07:56:00,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-07 07:56:00,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-07 07:56:00,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-07 07:56:00,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-07 07:56:00,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-07 07:56:00,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-07 07:56:00,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-07 07:56:00,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-07 07:56:00,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-07 07:56:00,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-07 07:56:00,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-07 07:56:00,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-07 07:56:00,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-07 07:56:00,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-07 07:56:00,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-07 07:56:00,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-07 07:56:00,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-07 07:56:00,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-07 07:56:00,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-07 07:56:00,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-07 07:56:00,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-07 07:56:00,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-07 07:56:00,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-07 07:56:00,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-07 07:56:00,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-07 07:56:00,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-07 07:56:00,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-07 07:56:00,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-07 07:56:00,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-07 07:56:00,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-07 07:56:00,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-07 07:56:00,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-07 07:56:00,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-07 07:56:00,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-07 07:56:00,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-07 07:56:00,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-07 07:56:00,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-07 07:56:00,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-07 07:56:00,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-07 07:56:00,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-07 07:56:00,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-07 07:56:00,288 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021257393834088253); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029132616006393625, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021257393834088253); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04516666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 07:56:00,289 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-07 07:56:00,289 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014652949578860618, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021257393834088253); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029132616006393625, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021257393834088253); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0115); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04516666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-07 07:56:00,555 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 353 ms to try 5000 different iterations.  Found a solution that moves 958 regions; Going from a computed imbalance of 0.014652949578860618 to a new imbalance of 0.0010656109472796365. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006880230611214945); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004544332072404845); MoveCostFunction : (multiplier=7.0, imbalance=0.10644444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006880230611214945); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=7.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 07:56:00,557 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1785602387:82 , srv1011607232:83 , srv124533164:83 , srv1287679772:83 , srv1355117153:83 , srv147355003:83 , srv1695587777:83 , srv1933765852:83 , srv2014227646:83 , srv366344324:83 , srv861602959:83 , srv1024465243:84 , srv1162142536:84 , srv1185984832:84 , srv1321175387:84 , srv1533074117:84 , srv1588807871:84 , srv1751270757:84 , srv1853516268:84 , srv1870930230:84 , srv1953549157:84 , srv2118049052:84 , srv2140446375:84 , srv736879647:84 , srv829031170:84 , srv885165423:84 , srv960023767:84 , srv122095055:85 , srv1283186923:85 , srv158676659:85 , srv1600615414:85 , srv1629740695:85 , srv1825520115:85 , srv1885789381:85 , srv1967210897:85 , srv197396424:85 , srv1978777344:85 , srv2031921526:85 , srv2039907837:85 , srv205451246:85 , srv2055266885:85 , srv2064569061:85 , srv326544333:85 , srv343132610:85 , srv431535873:85 , srv433922279:85 , srv586267026:85 , srv791263481:85 , srv812723856:85 , srv84218784:85 , srv867737394:85 , srv994816805:85 , srv998834441:85 , srv1159259820:86 , srv1290773933:86 , srv1377722616:86 , srv138178180:86 , srv1400322858:86 , srv1537143367:86 , srv1691499283:86 , srv1801323483:86 , srv1833794316:86 , srv1884071972:86 , srv1907636599:86 , srv1967619596:86 , srv2132411364:86 , srv239967283:86 , srv288609276:86 , srv300019186:86 , srv332990:86 , srv413288559:86 , srv449830008:86 , srv514152907:86 , srv525176650:86 , srv661354489:86 , srv851534272:86 , srv88401195:86 , srv885705143:86 , srv979036438:86 , srv1097422360:87 , srv115004759:87 , srv1221570247:87 , srv1225957302:87 , srv1240033938:87 , srv1475812271:87 , srv1544124103:87 , srv2029533346:87 , srv2091889995:87 , srv303934430:87 , srv366469408:87 , srv394109862:87 , srv54702485:87 , srv549669212:87 , srv770691415:87 , srv805231099:87 , srv828318618:87 , srv921643377:87 , srv1912837672:88 , srv577919360:88 , srv921035954:556 }
2024-03-07 07:56:00,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 07:56:00,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1537143367=23, srv1912837672=41, srv433922279=71, srv1801323483=33, srv1162142536=5, srv1695587777=30, srv138178180=18, srv770691415=81, srv1287679772=13, srv1240033938=10, srv829031170=86, srv885705143=93, srv1544124103=24, srv885165423=92, srv1825520115=34, srv805231099=83, srv1400322858=19, srv1967619596=45, srv1290773933=14, srv1691499283=29, srv994816805=98, srv1588807871=26, srv1283186923=12, srv1321175387=15, srv2039907837=51, srv828318618=85, srv851534272=88, srv147355003=20, srv2091889995=55, srv586267026=78, srv791263481=82, srv1097422360=2, srv1377722616=17, srv861602959=89, srv1870930230=37, srv736879647=80, srv2118049052=56, srv549669212=76, srv921035954=94, srv1629740695=28, srv300019186=61, srv326544333=63, srv1884071972=38, srv366344324=66, srv514152907=73, srv1221570247=8, srv1159259820=4, srv812723856=84, srv54702485=75, srv979036438=97, srv2014227646=48, srv2140446375=58, srv1024465243=1, srv115004759=3, srv2055266885=53, srv960023767=96, srv332990=64, srv1785602387=32, srv1600615414=27, srv525176650=74, srv2064569061=54, srv303934430=62, srv1853516268=36, srv124533164=11, srv413288559=69, srv205451246=52, srv1833794316=35, srv88401195=91, srv449830008=72, srv366469408=67, srv867737394=90, srv1953549157=43, srv998834441=99, srv2031921526=50, srv1933765852=42, srv84218784=87, srv1885789381=39, srv1355117153=16, srv394109862=68, srv2029533346=49, srv1978777344=47, srv2132411364=57, srv343132610=65, srv1751270757=31, srv239967283=59, srv1907636599=40, srv1225957302=9, srv197396424=46, srv288609276=60, srv1475812271=21, srv1185984832=6, srv122095055=7, srv431535873=70, srv661354489=79, srv1533074117=22, srv577919360=77, srv158676659=25, srv1967210897=44, srv921643377=95, srv1011607232=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 07:56:00,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 07:56:00,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 07:56:00,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 07:56:00,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 07:56:00,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 07:56:00,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 07:56:00,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 07:56:00,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 07:56:00,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 07:56:00,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 07:56:00,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 07:56:00,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 07:56:00,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 07:56:00,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 07:56:00,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 07:56:00,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 07:56:00,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 07:56:00,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 07:56:00,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 07:56:00,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 07:56:00,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 07:56:00,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 07:56:00,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 07:56:00,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 07:56:00,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 07:56:00,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 07:56:00,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 07:56:00,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 07:56:00,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 07:56:00,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 07:56:00,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-07 07:56:00,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-07 07:56:00,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-07 07:56:00,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-07 07:56:00,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-07 07:56:00,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-07 07:56:00,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-07 07:56:00,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-07 07:56:00,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-07 07:56:00,581 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-07 07:56:00,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-07 07:56:00,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-07 07:56:00,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-07 07:56:00,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-07 07:56:00,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-07 07:56:00,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-07 07:56:00,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-07 07:56:00,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-07 07:56:00,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-07 07:56:00,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-07 07:56:00,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-07 07:56:00,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-07 07:56:00,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-07 07:56:00,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-07 07:56:00,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-07 07:56:00,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-07 07:56:00,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-07 07:56:00,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-07 07:56:00,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-07 07:56:00,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-07 07:56:00,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-07 07:56:00,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-07 07:56:00,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-07 07:56:00,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-07 07:56:00,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-07 07:56:00,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-07 07:56:00,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-07 07:56:00,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-07 07:56:00,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-07 07:56:00,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-07 07:56:00,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-07 07:56:00,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-07 07:56:00,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-07 07:56:00,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-07 07:56:00,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-07 07:56:00,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-07 07:56:00,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-07 07:56:00,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-07 07:56:00,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-07 07:56:00,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-07 07:56:00,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-07 07:56:00,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-07 07:56:00,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-07 07:56:00,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-07 07:56:00,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-07 07:56:00,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-07 07:56:00,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-07 07:56:00,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-07 07:56:00,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-07 07:56:00,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-07 07:56:00,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-07 07:56:00,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-07 07:56:00,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-07 07:56:00,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-07 07:56:00,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-07 07:56:00,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-07 07:56:00,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-07 07:56:00,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-07 07:56:00,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-07 07:56:00,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-07 07:56:00,588 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-07 07:56:00,724 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006880230611214945); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004544332072404845); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006880230611214945); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=7.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 07:56:00,725 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-07 07:56:00,725 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0010589028879054087, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006880230611214945); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004544332072404845); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006880230611214945); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=7.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0036666666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-07 07:56:00,946 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 343 ms to try 5000 different iterations.  Found a solution that moves 225 regions; Going from a computed imbalance of 0.0010589028879054087 to a new imbalance of 2.613271481003072E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.025); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 07:56:00,947 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1825520115:83 , srv197396424:83 , srv2039907837:83 , srv1024465243:84 , srv1097422360:84 , srv1159259820:84 , srv1221570247:84 , srv1225957302:84 , srv1240033938:84 , srv1283186923:84 , srv1290773933:84 , srv1321175387:84 , srv1355117153:84 , srv1377722616:84 , srv138178180:84 , srv1400322858:84 , srv1537143367:84 , srv1544124103:84 , srv158676659:84 , srv1833794316:84 , srv1870930230:84 , srv1907636599:84 , srv1912837672:84 , srv1953549157:84 , srv1967619596:84 , srv1978777344:84 , srv2029533346:84 , srv300019186:84 , srv303934430:84 , srv366344324:84 , srv366469408:84 , srv413288559:84 , srv433922279:84 , srv525176650:84 , srv54702485:84 , srv577919360:84 , srv661354489:84 , srv736879647:84 , srv805231099:84 , srv812723856:84 , srv828318618:84 , srv829031170:84 , srv885705143:84 , srv921643377:84 , srv1011607232:85 , srv115004759:85 , srv1162142536:85 , srv124533164:85 , srv1287679772:85 , srv147355003:85 , srv1475812271:85 , srv1533074117:85 , srv1588807871:85 , srv1600615414:85 , srv1629740695:85 , srv1691499283:85 , srv1751270757:85 , srv1785602387:85 , srv1801323483:85 , srv1853516268:85 , srv1884071972:85 , srv1885789381:85 , srv1933765852:85 , srv1967210897:85 , srv2031921526:85 , srv205451246:85 , srv2055266885:85 , srv2064569061:85 , srv2091889995:85 , srv2118049052:85 , srv2132411364:85 , srv2140446375:85 , srv239967283:85 , srv288609276:85 , srv326544333:85 , srv332990:85 , srv343132610:85 , srv394109862:85 , srv449830008:85 , srv514152907:85 , srv549669212:85 , srv586267026:85 , srv791263481:85 , srv851534272:85 , srv861602959:85 , srv867737394:85 , srv885165423:85 , srv960023767:85 , srv979036438:85 , srv994816805:85 , srv998834441:85 , srv1185984832:86 , srv122095055:86 , srv1695587777:86 , srv2014227646:86 , srv431535873:86 , srv770691415:86 , srv88401195:86 , srv84218784:87 , srv921035954:623 }
2024-03-07 07:56:00,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 07:56:00,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1537143367=23, srv1912837672=41, srv433922279=71, srv1801323483=33, srv1162142536=5, srv1695587777=30, srv138178180=18, srv770691415=81, srv1287679772=13, srv1240033938=10, srv829031170=86, srv885705143=93, srv1544124103=24, srv885165423=92, srv1825520115=34, srv805231099=83, srv1400322858=19, srv1967619596=45, srv1290773933=14, srv1691499283=29, srv994816805=98, srv1588807871=26, srv1283186923=12, srv1321175387=15, srv2039907837=51, srv828318618=85, srv851534272=88, srv147355003=20, srv2091889995=55, srv586267026=78, srv791263481=82, srv1097422360=2, srv1377722616=17, srv861602959=89, srv1870930230=37, srv736879647=80, srv2118049052=56, srv549669212=76, srv921035954=94, srv1629740695=28, srv300019186=61, srv326544333=63, srv1884071972=38, srv366344324=66, srv514152907=73, srv1221570247=8, srv1159259820=4, srv812723856=84, srv54702485=75, srv979036438=97, srv2014227646=48, srv2140446375=58, srv1024465243=1, srv115004759=3, srv2055266885=53, srv960023767=96, srv332990=64, srv1785602387=32, srv1600615414=27, srv525176650=74, srv2064569061=54, srv303934430=62, srv1853516268=36, srv124533164=11, srv413288559=69, srv205451246=52, srv1833794316=35, srv88401195=91, srv449830008=72, srv366469408=67, srv867737394=90, srv1953549157=43, srv998834441=99, srv2031921526=50, srv1933765852=42, srv84218784=87, srv1885789381=39, srv1355117153=16, srv394109862=68, srv2029533346=49, srv1978777344=47, srv2132411364=57, srv343132610=65, srv1751270757=31, srv239967283=59, srv1907636599=40, srv1225957302=9, srv197396424=46, srv288609276=60, srv1475812271=21, srv1185984832=6, srv122095055=7, srv431535873=70, srv661354489=79, srv1533074117=22, srv577919360=77, srv158676659=25, srv1967210897=44, srv921643377=95, srv1011607232=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 07:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 07:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 07:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 07:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 07:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 07:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 07:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 07:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 07:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 07:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 07:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 07:56:00,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 07:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 07:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 07:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 07:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 07:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 07:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 07:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 07:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 07:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 07:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 07:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 07:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 07:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 07:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 07:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 07:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 07:56:00,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 07:56:00,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 07:56:00,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-07 07:56:00,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-07 07:56:00,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-07 07:56:00,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-07 07:56:00,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-07 07:56:00,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-07 07:56:00,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-07 07:56:00,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-07 07:56:00,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-07 07:56:00,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-07 07:56:00,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-07 07:56:00,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-07 07:56:00,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-07 07:56:00,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-07 07:56:00,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-07 07:56:00,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-07 07:56:00,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-07 07:56:00,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-07 07:56:00,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-07 07:56:00,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-07 07:56:00,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-07 07:56:00,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-07 07:56:00,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-07 07:56:00,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-07 07:56:00,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-07 07:56:00,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-07 07:56:00,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-07 07:56:00,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-07 07:56:00,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-07 07:56:00,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-07 07:56:00,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-07 07:56:00,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-07 07:56:00,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-07 07:56:00,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-07 07:56:00,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-07 07:56:00,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-07 07:56:00,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-07 07:56:00,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-07 07:56:00,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-07 07:56:00,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-07 07:56:00,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-07 07:56:00,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-07 07:56:00,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-07 07:56:00,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-07 07:56:00,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-07 07:56:00,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-07 07:56:00,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-07 07:56:00,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-07 07:56:00,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-07 07:56:00,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-07 07:56:00,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-07 07:56:00,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-07 07:56:00,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-07 07:56:00,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-07 07:56:00,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-07 07:56:00,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-07 07:56:00,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-07 07:56:00,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-07 07:56:00,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-07 07:56:00,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-07 07:56:00,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-07 07:56:00,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-07 07:56:00,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-07 07:56:00,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-07 07:56:00,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-07 07:56:00,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-07 07:56:00,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-07 07:56:00,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-07 07:56:00,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-07 07:56:00,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-07 07:56:00,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-07 07:56:01,067 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 07:56:01,067 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-07 07:56:01,068 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.455723113654296E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-07 07:56:01,256 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 278 ms to try 5000 different iterations.  Found a solution that moves 20 regions; Going from a computed imbalance of 2.455723113654296E-5 to a new imbalance of 6.88411204703086E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0022222222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 07:56:01,257 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2055266885:83 , srv343132610:83 , srv431535873:83 , srv770691415:83 , srv84218784:83 , srv851534272:83 , srv1011607232:84 , srv1024465243:84 , srv1097422360:84 , srv115004759:84 , srv1159259820:84 , srv1162142536:84 , srv1185984832:84 , srv122095055:84 , srv1221570247:84 , srv1225957302:84 , srv1240033938:84 , srv124533164:84 , srv1287679772:84 , srv1321175387:84 , srv1355117153:84 , srv1377722616:84 , srv138178180:84 , srv1400322858:84 , srv147355003:84 , srv1475812271:84 , srv1533074117:84 , srv1537143367:84 , srv1544124103:84 , srv158676659:84 , srv1588807871:84 , srv1600615414:84 , srv1629740695:84 , srv1691499283:84 , srv1695587777:84 , srv1751270757:84 , srv1785602387:84 , srv1801323483:84 , srv1825520115:84 , srv1833794316:84 , srv1853516268:84 , srv1870930230:84 , srv1884071972:84 , srv1885789381:84 , srv1907636599:84 , srv1912837672:84 , srv1933765852:84 , srv1967210897:84 , srv1967619596:84 , srv1978777344:84 , srv2014227646:84 , srv2029533346:84 , srv2031921526:84 , srv205451246:84 , srv2064569061:84 , srv2091889995:84 , srv2118049052:84 , srv2132411364:84 , srv2140446375:84 , srv239967283:84 , srv288609276:84 , srv300019186:84 , srv303934430:84 , srv326544333:84 , srv332990:84 , srv366344324:84 , srv366469408:84 , srv394109862:84 , srv413288559:84 , srv449830008:84 , srv514152907:84 , srv525176650:84 , srv54702485:84 , srv549669212:84 , srv577919360:84 , srv586267026:84 , srv661354489:84 , srv736879647:84 , srv791263481:84 , srv805231099:84 , srv812723856:84 , srv828318618:84 , srv829031170:84 , srv861602959:84 , srv867737394:84 , srv88401195:84 , srv885165423:84 , srv885705143:84 , srv921643377:84 , srv960023767:84 , srv979036438:84 , srv994816805:84 , srv998834441:84 , srv1283186923:85 , srv1290773933:85 , srv1953549157:85 , srv197396424:85 , srv2039907837:85 , srv433922279:85 , srv921035954:684 }
2024-03-07 07:56:01,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 07:56:01,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1537143367=23, srv1912837672=41, srv433922279=71, srv1801323483=33, srv1162142536=5, srv1695587777=30, srv138178180=18, srv770691415=81, srv1287679772=13, srv1240033938=10, srv829031170=86, srv885705143=93, srv1544124103=24, srv885165423=92, srv1825520115=34, srv805231099=83, srv1400322858=19, srv1967619596=45, srv1290773933=14, srv1691499283=29, srv994816805=98, srv1588807871=26, srv1283186923=12, srv1321175387=15, srv2039907837=51, srv828318618=85, srv851534272=88, srv147355003=20, srv2091889995=55, srv586267026=78, srv791263481=82, srv1097422360=2, srv1377722616=17, srv861602959=89, srv1870930230=37, srv736879647=80, srv2118049052=56, srv549669212=76, srv921035954=94, srv1629740695=28, srv300019186=61, srv326544333=63, srv1884071972=38, srv366344324=66, srv514152907=73, srv1221570247=8, srv1159259820=4, srv812723856=84, srv54702485=75, srv979036438=97, srv2014227646=48, srv2140446375=58, srv1024465243=1, srv115004759=3, srv2055266885=53, srv960023767=96, srv332990=64, srv1785602387=32, srv1600615414=27, srv525176650=74, srv2064569061=54, srv303934430=62, srv1853516268=36, srv124533164=11, srv413288559=69, srv205451246=52, srv1833794316=35, srv88401195=91, srv449830008=72, srv366469408=67, srv867737394=90, srv1953549157=43, srv998834441=99, srv2031921526=50, srv1933765852=42, srv84218784=87, srv1885789381=39, srv1355117153=16, srv394109862=68, srv2029533346=49, srv1978777344=47, srv2132411364=57, srv343132610=65, srv1751270757=31, srv239967283=59, srv1907636599=40, srv1225957302=9, srv197396424=46, srv288609276=60, srv1475812271=21, srv1185984832=6, srv122095055=7, srv431535873=70, srv661354489=79, srv1533074117=22, srv577919360=77, srv158676659=25, srv1967210897=44, srv921643377=95, srv1011607232=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 07:56:01,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 07:56:01,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 07:56:01,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 07:56:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 07:56:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 07:56:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 07:56:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 07:56:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 07:56:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 07:56:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 07:56:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 07:56:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 07:56:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 07:56:01,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 07:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 07:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 07:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 07:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 07:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 07:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 07:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 07:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 07:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 07:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 07:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 07:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 07:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 07:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 07:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 07:56:01,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 07:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-07 07:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-07 07:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-07 07:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-07 07:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-07 07:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-07 07:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-07 07:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-07 07:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-07 07:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-07 07:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-07 07:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-07 07:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-07 07:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-07 07:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-07 07:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-07 07:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-07 07:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-07 07:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-07 07:56:01,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-07 07:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-07 07:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-07 07:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-07 07:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-07 07:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-07 07:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-07 07:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-07 07:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-07 07:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-07 07:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-07 07:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-07 07:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-07 07:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-07 07:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-07 07:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-07 07:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-07 07:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-07 07:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-07 07:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-07 07:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-07 07:56:01,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-07 07:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-07 07:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-07 07:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-07 07:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-07 07:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-07 07:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-07 07:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-07 07:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-07 07:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-07 07:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-07 07:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-07 07:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-07 07:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-07 07:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-07 07:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-07 07:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-07 07:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-07 07:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-07 07:56:01,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-07 07:56:01,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-07 07:56:01,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-07 07:56:01,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-07 07:56:01,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-07 07:56:01,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-07 07:56:01,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-07 07:56:01,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-07 07:56:01,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-07 07:56:01,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-07 07:56:01,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-07 07:56:01,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-07 07:56:01,367 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 07:56:01,368 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv2055266885:83 , srv343132610:83 , srv431535873:83 , srv770691415:83 , srv84218784:83 , srv851534272:83 , srv1011607232:84 , srv1024465243:84 , srv1097422360:84 , srv115004759:84 , srv1159259820:84 , srv1162142536:84 , srv1185984832:84 , srv122095055:84 , srv1221570247:84 , srv1225957302:84 , srv1240033938:84 , srv124533164:84 , srv1287679772:84 , srv1321175387:84 , srv1355117153:84 , srv1377722616:84 , srv138178180:84 , srv1400322858:84 , srv147355003:84 , srv1475812271:84 , srv1533074117:84 , srv1537143367:84 , srv1544124103:84 , srv158676659:84 , srv1588807871:84 , srv1600615414:84 , srv1629740695:84 , srv1691499283:84 , srv1695587777:84 , srv1751270757:84 , srv1785602387:84 , srv1801323483:84 , srv1825520115:84 , srv1833794316:84 , srv1853516268:84 , srv1870930230:84 , srv1884071972:84 , srv1885789381:84 , srv1907636599:84 , srv1912837672:84 , srv1933765852:84 , srv1967210897:84 , srv1967619596:84 , srv1978777344:84 , srv2014227646:84 , srv2029533346:84 , srv2031921526:84 , srv205451246:84 , srv2064569061:84 , srv2091889995:84 , srv2118049052:84 , srv2132411364:84 , srv2140446375:84 , srv239967283:84 , srv288609276:84 , srv300019186:84 , srv303934430:84 , srv326544333:84 , srv332990:84 , srv366344324:84 , srv366469408:84 , srv394109862:84 , srv413288559:84 , srv449830008:84 , srv514152907:84 , srv525176650:84 , srv54702485:84 , srv549669212:84 , srv577919360:84 , srv586267026:84 , srv661354489:84 , srv736879647:84 , srv791263481:84 , srv805231099:84 , srv812723856:84 , srv828318618:84 , srv829031170:84 , srv861602959:84 , srv867737394:84 , srv88401195:84 , srv885165423:84 , srv885705143:84 , srv921643377:84 , srv960023767:84 , srv979036438:84 , srv994816805:84 , srv998834441:84 , srv1283186923:85 , srv1290773933:85 , srv1953549157:85 , srv197396424:85 , srv2039907837:85 , srv433922279:85 , srv921035954:684 }
2024-03-07 07:56:01,401 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=198 (was 190) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=1537 (was 2072)
2024-03-07 07:56:01,413 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=198, ProcessCount=168, AvailableMemoryMB=1537
2024-03-07 07:56:01,414 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-07 07:56:01,414 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-03-07 07:56:01,424 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1088332288:84 , srv114989564:84 , srv1173618793:84 , srv1196635605:84 , srv1205275675:84 , srv1241498301:84 , srv1273942787:84 , srv1350090266:84 , srv1434719708:84 , srv1610232513:84 , srv1613410709:84 , srv1615893210:84 , srv1721237341:84 , srv1727668348:84 , srv1807637959:84 , srv1860587897:84 , srv2029976854:84 , srv2042230203:84 , srv2065400194:84 , srv2131141136:84 , srv236848498:84 , srv333315106:84 , srv385246540:84 , srv466027134:84 , srv474686939:84 , srv521614505:84 , srv748361288:84 , srv76806137:84 , srv97852956:84 , srv1385727647:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-03-07 07:56:01,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 07:56:01,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1241498301=5, srv1350090266=7, srv1205275675=4, srv2042230203=18, srv385246540=23, srv1860587897=16, srv2029976854=17, srv748361288=27, srv1807637959=15, srv1615893210=12, srv333315106=22, srv1273942787=6, srv1613410709=11, srv76806137=28, srv236848498=21, srv1196635605=3, srv1721237341=13, srv97852956=29, srv114989564=1, srv1385727647=8, srv466027134=24, srv2065400194=19, srv1434719708=9, srv1088332288=0, srv2131141136=20, srv1727668348=14, srv521614505=26, srv474686939=25, srv1610232513=10, srv1173618793=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 07:56:01,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 07:56:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 07:56:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 07:56:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 07:56:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 07:56:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 07:56:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 07:56:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 07:56:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 07:56:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 07:56:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 07:56:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 07:56:01,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 07:56:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 07:56:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 07:56:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 07:56:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 07:56:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 07:56:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 07:56:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 07:56:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 07:56:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 07:56:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 07:56:01,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 07:56:01,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 07:56:01,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 07:56:01,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 07:56:01,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 07:56:01,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 07:56:01,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 07:56:01,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-07 07:56:01,467 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0763908324811846, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 07:56:01,468 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912433748283038, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0763908324811846, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-03-07 07:58:01,448 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9881579 different iterations.  Found a solution that moves 1833 regions; Going from a computed imbalance of 0.9912433748283038 to a new imbalance of 4.278313442226764E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6788888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 07:58:01,453 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1088332288:90 , srv114989564:90 , srv1173618793:90 , srv1196635605:90 , srv1205275675:90 , srv1241498301:90 , srv1273942787:90 , srv1350090266:90 , srv1385727647:90 , srv1434719708:90 , srv1610232513:90 , srv1613410709:90 , srv1615893210:90 , srv1721237341:90 , srv1727668348:90 , srv1807637959:90 , srv1860587897:90 , srv2029976854:90 , srv2042230203:90 , srv2065400194:90 , srv2131141136:90 , srv236848498:90 , srv333315106:90 , srv385246540:90 , srv466027134:90 , srv474686939:90 , srv521614505:90 , srv748361288:90 , srv76806137:90 , srv97852956:90 }
2024-03-07 07:58:01,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-07 07:58:01,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1241498301=5, srv1350090266=7, srv1205275675=4, srv2042230203=18, srv385246540=23, srv1860587897=16, srv2029976854=17, srv748361288=27, srv1807637959=15, srv1615893210=12, srv333315106=22, srv1273942787=6, srv1613410709=11, srv76806137=28, srv236848498=21, srv1196635605=3, srv1721237341=13, srv97852956=29, srv114989564=1, srv1385727647=8, srv466027134=24, srv2065400194=19, srv1434719708=9, srv1088332288=0, srv2131141136=20, srv1727668348=14, srv521614505=26, srv474686939=25, srv1610232513=10, srv1173618793=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-07 07:58:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-07 07:58:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-07 07:58:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-07 07:58:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-07 07:58:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-07 07:58:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-07 07:58:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-07 07:58:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-07 07:58:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-07 07:58:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-07 07:58:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-07 07:58:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-07 07:58:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-07 07:58:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-07 07:58:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-07 07:58:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-07 07:58:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-07 07:58:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-07 07:58:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-07 07:58:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-07 07:58:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-07 07:58:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-07 07:58:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-07 07:58:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-07 07:58:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-07 07:58:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-07 07:58:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-07 07:58:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-07 07:58:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-07 07:58:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-07 07:58:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-07 07:58:01,468 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-07 07:58:01,468 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1088332288:90 , srv114989564:90 , srv1173618793:90 , srv1196635605:90 , srv1205275675:90 , srv1241498301:90 , srv1273942787:90 , srv1350090266:90 , srv1385727647:90 , srv1434719708:90 , srv1610232513:90 , srv1613410709:90 , srv1615893210:90 , srv1721237341:90 , srv1727668348:90 , srv1807637959:90 , srv1860587897:90 , srv2029976854:90 , srv2042230203:90 , srv2065400194:90 , srv2131141136:90 , srv236848498:90 , srv333315106:90 , srv385246540:90 , srv466027134:90 , srv474686939:90 , srv521614505:90 , srv748361288:90 , srv76806137:90 , srv97852956:90 }
2024-03-07 07:58:01,480 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=118 (was 198), ProcessCount=165 (was 168), AvailableMemoryMB=2786 (was 1537) - AvailableMemoryMB LEAK? -