2023-10-14 13:57:11,389 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-10-14 13:57:11,642 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-10-14 13:57:11,729 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-10-14 13:57:11,787 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-10-14 13:57:11,792 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-10-14 13:57:11,803 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=266, ProcessCount=168, AvailableMemoryMB=4218
2023-10-14 13:57:11,805 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-10-14 13:57:11,805 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-10-14 13:57:11,930 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1006584719:84 , srv1006590209:84 , srv1008554496:84 , srv1028208265:84 , srv1098507372:84 , srv109943941:84 , srv1109624120:84 , srv1127776738:84 , srv1129189678:84 , srv12615721:84 , srv1270692772:84 , srv127138438:84 , srv1303204096:84 , srv1311037666:84 , srv1318942747:84 , srv1325259187:84 , srv1338509685:84 , srv133935008:84 , srv1362069347:84 , srv1397901949:84 , srv1413034191:84 , srv1436699060:84 , srv1452142038:84 , srv1476096140:84 , srv1477968373:84 , srv149830880:84 , srv1503102391:84 , srv1507956776:84 , srv1513128483:84 , srv1528550309:84 , srv1539683888:84 , srv154443427:84 , srv1550443171:84 , srv1555255317:84 , srv1585016081:84 , srv1606396430:84 , srv1618314271:84 , srv162504894:84 , srv1628320217:84 , srv1644471815:84 , srv1650918008:84 , srv1662167050:84 , srv168981059:84 , srv1693265536:84 , srv1706228640:84 , srv171349333:84 , srv1727575309:84 , srv1741322673:84 , srv1749379941:84 , srv1792292002:84 , srv1804946621:84 , srv1816687253:84 , srv1821216564:84 , srv1839816345:84 , srv1895483566:84 , srv1910726455:84 , srv1913320832:84 , srv1923061313:84 , srv1942293077:84 , srv1943236897:84 , srv1990602888:84 , srv200085113:84 , srv2027129669:84 , srv2039131977:84 , srv2044321696:84 , srv2045124652:84 , srv2071209181:84 , srv2091283812:84 , srv21003249:84 , srv2132601213:84 , srv224156068:84 , srv232983289:84 , srv263947563:84 , srv301362638:84 , srv303662032:84 , srv326417666:84 , srv36922699:84 , srv384678811:84 , srv430089558:84 , srv432452400:84 , srv437736810:84 , srv445574129:84 , srv486632795:84 , srv543844129:84 , srv545518205:84 , srv579555538:84 , srv582724493:84 , srv589780314:84 , srv630128150:84 , srv645317315:84 , srv648229297:84 , srv727172727:84 , srv737414076:84 , srv830475214:84 , srv861238598:84 , srv866015008:84 , srv868331236:84 , srv876356254:84 , srv877411892:84 , srv1120316696:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-10-14 13:57:11,942 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-14 13:57:11,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv430089558=79, srv1706228640=45, srv326417666=76, srv432452400=80, srv12615721=10, srv133935008=18, srv2071209181=67, srv1507956776=28, srv876356254=98, srv1606396430=36, srv589780314=88, srv543844129=84, srv384678811=78, srv645317315=90, srv2045124652=66, srv168981059=43, srv582724493=87, srv830475214=94, srv1539683888=31, srv109943941=5, srv1270692772=11, srv1555255317=34, srv2044321696=65, srv1325259187=16, srv1028208265=3, srv1943236897=60, srv2027129669=63, srv1006584719=0, srv171349333=46, srv868331236=97, srv1821216564=53, srv1618314271=37, srv1644471815=40, srv1127776738=8, srv1129189678=9, srv149830880=26, srv1839816345=54, srv232983289=72, srv1816687253=52, srv445574129=82, srv2091283812=68, srv1804946621=51, srv1528550309=30, srv1477968373=25, srv1990602888=61, srv301362638=74, srv1749379941=49, srv1585016081=35, srv1650918008=41, srv727172727=92, srv1693265536=44, srv861238598=95, srv866015008=96, srv1452142038=23, srv1338509685=17, srv1727575309=47, srv36922699=77, srv1476096140=24, srv1397901949=20, srv1513128483=29, srv1318942747=15, srv877411892=99, srv162504894=38, srv127138438=12, srv1006590209=1, srv2039131977=64, srv545518205=85, srv21003249=69, srv200085113=62, srv263947563=73, srv486632795=83, srv1362069347=19, srv1628320217=39, srv1895483566=55, srv224156068=71, srv1913320832=57, srv437736810=81, srv1413034191=21, srv1303204096=13, srv1741322673=48, srv1942293077=59, srv737414076=93, srv1910726455=56, srv303662032=75, srv1120316696=7, srv1109624120=6, srv1923061313=58, srv1008554496=2, srv1311037666=14, srv2132601213=70, srv630128150=89, srv1503102391=27, srv579555538=86, srv648229297=91, srv1792292002=50, srv1098507372=4, srv154443427=32, srv1436699060=22, srv1662167050=42, srv1550443171=33} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-14 13:57:12,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-14 13:57:12,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-14 13:57:12,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-14 13:57:12,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-14 13:57:12,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-14 13:57:12,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-14 13:57:12,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-14 13:57:12,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-14 13:57:12,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-14 13:57:12,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-14 13:57:12,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-14 13:57:12,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-14 13:57:12,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-14 13:57:12,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-14 13:57:12,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-14 13:57:12,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-14 13:57:12,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-14 13:57:12,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-14 13:57:12,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-14 13:57:12,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-14 13:57:12,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-14 13:57:12,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-14 13:57:12,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-14 13:57:12,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-14 13:57:12,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-14 13:57:12,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-14 13:57:12,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-14 13:57:12,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-14 13:57:12,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-14 13:57:12,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-14 13:57:12,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-14 13:57:12,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-14 13:57:12,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-14 13:57:12,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-14 13:57:12,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-14 13:57:12,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-14 13:57:12,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-14 13:57:12,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-14 13:57:12,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-14 13:57:12,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-14 13:57:12,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-14 13:57:12,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-14 13:57:12,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-14 13:57:12,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-14 13:57:12,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-14 13:57:12,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-14 13:57:12,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-14 13:57:12,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-14 13:57:12,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-14 13:57:12,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-14 13:57:12,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-14 13:57:12,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-14 13:57:12,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-14 13:57:12,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-14 13:57:12,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-14 13:57:12,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-14 13:57:12,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-14 13:57:12,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-14 13:57:12,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-14 13:57:12,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-14 13:57:12,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-14 13:57:12,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-14 13:57:12,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-14 13:57:12,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-14 13:57:12,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-14 13:57:12,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-14 13:57:12,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-14 13:57:12,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-14 13:57:12,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-14 13:57:12,055 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-14 13:57:12,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-14 13:57:12,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-14 13:57:12,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-14 13:57:12,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-14 13:57:12,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-14 13:57:12,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-14 13:57:12,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-14 13:57:12,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-14 13:57:12,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-14 13:57:12,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-14 13:57:12,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-14 13:57:12,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-14 13:57:12,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-14 13:57:12,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-14 13:57:12,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-14 13:57:12,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-14 13:57:12,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-14 13:57:12,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-14 13:57:12,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-14 13:57:12,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-14 13:57:12,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-14 13:57:12,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-14 13:57:12,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-14 13:57:12,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-14 13:57:12,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-14 13:57:12,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-14 13:57:12,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-14 13:57:12,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-14 13:57:12,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-14 13:57:12,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-14 13:57:12,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-14 13:57:12,329 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-14 13:57:12,329 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-14 13:57:12,330 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-14 13:57:12,692 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 615 ms to try 5000 different iterations.  Found a solution that moves 4638 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18297719488228348. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04040263746161375, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.058670263128207535, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5153333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04040263746161375, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17841666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24283333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-14 13:57:12,702 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv301362638:81 , srv727172727:81 , srv1006590209:82 , srv1910726455:82 , srv1362069347:83 , srv171349333:83 , srv1741322673:83 , srv1839816345:83 , srv36922699:83 , srv430089558:83 , srv432452400:83 , srv589780314:83 , srv1098507372:84 , srv1127776738:84 , srv1129189678:84 , srv1270692772:84 , srv1476096140:84 , srv1662167050:84 , srv1693265536:84 , srv1816687253:84 , srv384678811:84 , srv445574129:84 , srv630128150:84 , srv645317315:84 , srv737414076:84 , srv877411892:84 , srv1028208265:85 , srv1452142038:85 , srv1477968373:85 , srv1528550309:85 , srv1618314271:85 , srv1644471815:85 , srv1706228640:85 , srv2044321696:85 , srv303662032:85 , srv326417666:85 , srv866015008:85 , srv1006584719:86 , srv109943941:86 , srv1109624120:86 , srv133935008:86 , srv1413034191:86 , srv1507956776:86 , srv1513128483:86 , srv1550443171:86 , srv1628320217:86 , srv1650918008:86 , srv1792292002:86 , srv1923061313:86 , srv200085113:86 , srv2027129669:86 , srv2091283812:86 , srv2132601213:86 , srv224156068:86 , srv263947563:86 , srv543844129:86 , srv582724493:86 , srv830475214:86 , srv868331236:86 , srv12615721:87 , srv1338509685:87 , srv149830880:87 , srv1804946621:87 , srv1895483566:87 , srv21003249:87 , srv545518205:87 , srv648229297:87 , srv127138438:88 , srv1397901949:88 , srv1503102391:88 , srv154443427:88 , srv1585016081:88 , srv1606396430:88 , srv162504894:88 , srv1821216564:88 , srv1913320832:88 , srv1943236897:88 , srv1990602888:88 , srv2071209181:88 , srv232983289:88 , srv437736810:88 , srv579555538:88 , srv876356254:88 , srv1325259187:89 , srv1555255317:89 , srv1727575309:89 , srv1749379941:89 , srv1008554496:90 , srv1303204096:90 , srv1436699060:90 , srv1942293077:90 , srv861238598:90 , srv168981059:91 , srv2039131977:91 , srv1318942747:92 , srv1539683888:92 , srv2045124652:92 , srv486632795:94 , srv1311037666:96 , srv1120316696:449 }
2023-10-14 13:57:12,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-14 13:57:12,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv430089558=79, srv1706228640=45, srv326417666=76, srv432452400=80, srv12615721=10, srv133935008=18, srv2071209181=67, srv1507956776=28, srv876356254=98, srv1606396430=36, srv589780314=88, srv543844129=84, srv384678811=78, srv645317315=90, srv2045124652=66, srv168981059=43, srv582724493=87, srv830475214=94, srv1539683888=31, srv109943941=5, srv1270692772=11, srv1555255317=34, srv2044321696=65, srv1325259187=16, srv1028208265=3, srv1943236897=60, srv2027129669=63, srv1006584719=0, srv171349333=46, srv868331236=97, srv1821216564=53, srv1618314271=37, srv1644471815=40, srv1127776738=8, srv1129189678=9, srv149830880=26, srv1839816345=54, srv232983289=72, srv1816687253=52, srv445574129=82, srv2091283812=68, srv1804946621=51, srv1528550309=30, srv1477968373=25, srv1990602888=61, srv301362638=74, srv1749379941=49, srv1585016081=35, srv1650918008=41, srv727172727=92, srv1693265536=44, srv861238598=95, srv866015008=96, srv1452142038=23, srv1338509685=17, srv1727575309=47, srv36922699=77, srv1476096140=24, srv1397901949=20, srv1513128483=29, srv1318942747=15, srv877411892=99, srv162504894=38, srv127138438=12, srv1006590209=1, srv2039131977=64, srv545518205=85, srv21003249=69, srv200085113=62, srv263947563=73, srv486632795=83, srv1362069347=19, srv1628320217=39, srv1895483566=55, srv224156068=71, srv1913320832=57, srv437736810=81, srv1413034191=21, srv1303204096=13, srv1741322673=48, srv1942293077=59, srv737414076=93, srv1910726455=56, srv303662032=75, srv1120316696=7, srv1109624120=6, srv1923061313=58, srv1008554496=2, srv1311037666=14, srv2132601213=70, srv630128150=89, srv1503102391=27, srv579555538=86, srv648229297=91, srv1792292002=50, srv1098507372=4, srv154443427=32, srv1436699060=22, srv1662167050=42, srv1550443171=33} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-14 13:57:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-14 13:57:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-14 13:57:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-14 13:57:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-14 13:57:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-14 13:57:12,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-14 13:57:12,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-14 13:57:12,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-14 13:57:12,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-14 13:57:12,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-14 13:57:12,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-14 13:57:12,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-14 13:57:12,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-14 13:57:12,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-14 13:57:12,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-14 13:57:12,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-14 13:57:12,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-14 13:57:12,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-14 13:57:12,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-14 13:57:12,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-14 13:57:12,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-14 13:57:12,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-14 13:57:12,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-14 13:57:12,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-14 13:57:12,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-14 13:57:12,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-14 13:57:12,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-14 13:57:12,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-14 13:57:12,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-14 13:57:12,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-14 13:57:12,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-14 13:57:12,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-14 13:57:12,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-14 13:57:12,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-14 13:57:12,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-14 13:57:12,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-14 13:57:12,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-14 13:57:12,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-14 13:57:12,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-14 13:57:12,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-14 13:57:12,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-14 13:57:12,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-14 13:57:12,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-14 13:57:12,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-14 13:57:12,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-14 13:57:12,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-14 13:57:12,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-14 13:57:12,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-14 13:57:12,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-14 13:57:12,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-14 13:57:12,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-14 13:57:12,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-14 13:57:12,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-14 13:57:12,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-14 13:57:12,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-14 13:57:12,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-14 13:57:12,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-14 13:57:12,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-14 13:57:12,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-14 13:57:12,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-14 13:57:12,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-14 13:57:12,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-14 13:57:12,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-14 13:57:12,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-14 13:57:12,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-14 13:57:12,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-14 13:57:12,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-14 13:57:12,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-14 13:57:12,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-14 13:57:12,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-14 13:57:12,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-14 13:57:12,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-14 13:57:12,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-14 13:57:12,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-14 13:57:12,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-14 13:57:12,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-14 13:57:12,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-14 13:57:12,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-14 13:57:12,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-14 13:57:12,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-14 13:57:12,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-14 13:57:12,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-14 13:57:12,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-14 13:57:12,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-14 13:57:12,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-14 13:57:12,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-14 13:57:12,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-14 13:57:12,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-14 13:57:12,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-14 13:57:12,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-14 13:57:12,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-14 13:57:12,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-14 13:57:12,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-14 13:57:12,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-14 13:57:12,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-14 13:57:12,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-14 13:57:12,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-14 13:57:12,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-14 13:57:12,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-14 13:57:12,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-14 13:57:12,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-14 13:57:12,837 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04040263746161375, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.058670263128207535, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04040263746161375, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17841666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24283333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-14 13:57:12,837 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-14 13:57:12,837 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18294471891216063, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04040263746161375, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.058670263128207535, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04040263746161375, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17841666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24283333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-14 13:57:13,027 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 274 ms to try 5000 different iterations.  Found a solution that moves 2680 regions; Going from a computed imbalance of 0.18294471891216063 to a new imbalance of 0.015890438512486355. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020928086059956226); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030229202996518344, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29777777777777775, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020928086059956226); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04616666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-14 13:57:13,038 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1311037666:75 , srv1539683888:78 , srv486632795:79 , srv2045124652:80 , srv1318942747:81 , srv1436699060:81 , srv1942293077:81 , srv861238598:81 , srv1303204096:82 , srv1727575309:82 , srv1943236897:82 , srv12615721:83 , srv1325259187:83 , srv1555255317:83 , srv168981059:83 , srv1821216564:83 , srv232983289:83 , srv437736810:83 , srv1008554496:84 , srv127138438:84 , srv1338509685:84 , srv149830880:84 , srv1503102391:84 , srv154443427:84 , srv1585016081:84 , srv1749379941:84 , srv1804946621:84 , srv1895483566:84 , srv21003249:84 , srv648229297:84 , srv876356254:84 , srv1109624120:85 , srv1507956776:85 , srv1606396430:85 , srv162504894:85 , srv1628320217:85 , srv1662167050:85 , srv1792292002:85 , srv1913320832:85 , srv1990602888:85 , srv2039131977:85 , srv2091283812:85 , srv263947563:85 , srv579555538:85 , srv1028208265:86 , srv109943941:86 , srv1127776738:86 , srv1413034191:86 , srv1452142038:86 , srv1513128483:86 , srv1550443171:86 , srv1706228640:86 , srv2132601213:86 , srv224156068:86 , srv303662032:86 , srv543844129:86 , srv545518205:86 , srv630128150:86 , srv830475214:86 , srv868331236:86 , srv1006584719:87 , srv1270692772:87 , srv1397901949:87 , srv1618314271:87 , srv1650918008:87 , srv1816687253:87 , srv2027129669:87 , srv2071209181:87 , srv326417666:87 , srv384678811:87 , srv582724493:87 , srv589780314:87 , srv737414076:87 , srv866015008:87 , srv877411892:87 , srv133935008:88 , srv1477968373:88 , srv1528550309:88 , srv1644471815:88 , srv1923061313:88 , srv200085113:88 , srv2044321696:88 , srv36922699:88 , srv445574129:88 , srv1129189678:89 , srv1362069347:89 , srv432452400:89 , srv1098507372:90 , srv1476096140:90 , srv1693265536:90 , srv1741322673:90 , srv1910726455:90 , srv301362638:90 , srv645317315:90 , srv171349333:91 , srv1839816345:91 , srv430089558:91 , srv1006590209:92 , srv727172727:94 , srv1120316696:511 }
2023-10-14 13:57:13,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-14 13:57:13,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv430089558=79, srv1706228640=45, srv326417666=76, srv432452400=80, srv12615721=10, srv133935008=18, srv2071209181=67, srv1507956776=28, srv876356254=98, srv1606396430=36, srv589780314=88, srv543844129=84, srv384678811=78, srv645317315=90, srv2045124652=66, srv168981059=43, srv582724493=87, srv830475214=94, srv1539683888=31, srv109943941=5, srv1270692772=11, srv1555255317=34, srv2044321696=65, srv1325259187=16, srv1028208265=3, srv1943236897=60, srv2027129669=63, srv1006584719=0, srv171349333=46, srv868331236=97, srv1821216564=53, srv1618314271=37, srv1644471815=40, srv1127776738=8, srv1129189678=9, srv149830880=26, srv1839816345=54, srv232983289=72, srv1816687253=52, srv445574129=82, srv2091283812=68, srv1804946621=51, srv1528550309=30, srv1477968373=25, srv1990602888=61, srv301362638=74, srv1749379941=49, srv1585016081=35, srv1650918008=41, srv727172727=92, srv1693265536=44, srv861238598=95, srv866015008=96, srv1452142038=23, srv1338509685=17, srv1727575309=47, srv36922699=77, srv1476096140=24, srv1397901949=20, srv1513128483=29, srv1318942747=15, srv877411892=99, srv162504894=38, srv127138438=12, srv1006590209=1, srv2039131977=64, srv545518205=85, srv21003249=69, srv200085113=62, srv263947563=73, srv486632795=83, srv1362069347=19, srv1628320217=39, srv1895483566=55, srv224156068=71, srv1913320832=57, srv437736810=81, srv1413034191=21, srv1303204096=13, srv1741322673=48, srv1942293077=59, srv737414076=93, srv1910726455=56, srv303662032=75, srv1120316696=7, srv1109624120=6, srv1923061313=58, srv1008554496=2, srv1311037666=14, srv2132601213=70, srv630128150=89, srv1503102391=27, srv579555538=86, srv648229297=91, srv1792292002=50, srv1098507372=4, srv154443427=32, srv1436699060=22, srv1662167050=42, srv1550443171=33} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-14 13:57:13,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-14 13:57:13,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-14 13:57:13,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-14 13:57:13,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-14 13:57:13,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-14 13:57:13,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-14 13:57:13,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-14 13:57:13,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-14 13:57:13,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-14 13:57:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-14 13:57:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-14 13:57:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-14 13:57:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-14 13:57:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-14 13:57:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-14 13:57:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-14 13:57:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-14 13:57:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-14 13:57:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-14 13:57:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-14 13:57:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-14 13:57:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-14 13:57:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-14 13:57:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-14 13:57:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-14 13:57:13,058 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-14 13:57:13,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-14 13:57:13,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-14 13:57:13,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-14 13:57:13,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-14 13:57:13,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-14 13:57:13,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-14 13:57:13,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-14 13:57:13,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-14 13:57:13,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-14 13:57:13,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-14 13:57:13,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-14 13:57:13,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-14 13:57:13,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-14 13:57:13,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-14 13:57:13,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-14 13:57:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-14 13:57:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-14 13:57:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-14 13:57:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-14 13:57:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-14 13:57:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-14 13:57:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-14 13:57:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-14 13:57:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-14 13:57:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-14 13:57:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-14 13:57:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-14 13:57:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-14 13:57:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-14 13:57:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-14 13:57:13,060 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-14 13:57:13,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-14 13:57:13,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-14 13:57:13,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-14 13:57:13,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-14 13:57:13,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-14 13:57:13,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-14 13:57:13,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-14 13:57:13,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-14 13:57:13,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-14 13:57:13,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-14 13:57:13,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-14 13:57:13,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-14 13:57:13,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-14 13:57:13,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-14 13:57:13,061 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-14 13:57:13,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-14 13:57:13,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-14 13:57:13,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-14 13:57:13,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-14 13:57:13,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-14 13:57:13,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-14 13:57:13,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-14 13:57:13,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-14 13:57:13,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-14 13:57:13,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-14 13:57:13,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-14 13:57:13,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-14 13:57:13,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-14 13:57:13,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-14 13:57:13,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-14 13:57:13,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-14 13:57:13,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-14 13:57:13,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-14 13:57:13,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-14 13:57:13,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-14 13:57:13,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-14 13:57:13,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-14 13:57:13,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-14 13:57:13,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-14 13:57:13,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-14 13:57:13,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-14 13:57:13,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-14 13:57:13,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-14 13:57:13,063 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-14 13:57:13,154 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020928086059956226); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030229202996518344, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020928086059956226); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04616666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-14 13:57:13,154 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-14 13:57:13,155 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0158716727513977, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020928086059956226); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.030229202996518344, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020928086059956226); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04616666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-14 13:57:13,298 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 223 ms to try 5000 different iterations.  Found a solution that moves 993 regions; Going from a computed imbalance of 0.0158716727513977 to a new imbalance of 6.609420326514888E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00365626180754161); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006374049520922626); MoveCostFunction : (multiplier=7.0, imbalance=0.11033333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00365626180754161); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-14 13:57:13,300 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2039131977:81 , srv1397901949:83 , srv1476096140:83 , srv1693265536:83 , srv171349333:83 , srv727172727:83 , srv1006584719:84 , srv1006590209:84 , srv1008554496:84 , srv1098507372:84 , srv1129189678:84 , srv133935008:84 , srv1528550309:84 , srv162504894:84 , srv1650918008:84 , srv1741322673:84 , srv1839816345:84 , srv1913320832:84 , srv200085113:84 , srv2027129669:84 , srv2071209181:84 , srv430089558:84 , srv486632795:84 , srv545518205:84 , srv582724493:84 , srv645317315:84 , srv109943941:85 , srv1303204096:85 , srv1318942747:85 , srv1362069347:85 , srv1477968373:85 , srv1503102391:85 , srv1513128483:85 , srv154443427:85 , srv1555255317:85 , srv1606396430:85 , srv1618314271:85 , srv1644471815:85 , srv168981059:85 , srv1749379941:85 , srv1923061313:85 , srv1990602888:85 , srv2044321696:85 , srv224156068:85 , srv326417666:85 , srv432452400:85 , srv445574129:85 , srv579555538:85 , srv866015008:85 , srv127138438:86 , srv1325259187:86 , srv1338509685:86 , srv1550443171:86 , srv1585016081:86 , srv1628320217:86 , srv1727575309:86 , srv1821216564:86 , srv1910726455:86 , srv2045124652:86 , srv2132601213:86 , srv232983289:86 , srv36922699:86 , srv384678811:86 , srv543844129:86 , srv737414076:86 , srv830475214:86 , srv861238598:86 , srv868331236:86 , srv876356254:86 , srv877411892:86 , srv1028208265:87 , srv1109624120:87 , srv1127776738:87 , srv1270692772:87 , srv1311037666:87 , srv1436699060:87 , srv1452142038:87 , srv149830880:87 , srv1507956776:87 , srv1792292002:87 , srv1804946621:87 , srv1816687253:87 , srv1895483566:87 , srv1942293077:87 , srv2091283812:87 , srv21003249:87 , srv263947563:87 , srv303662032:87 , srv437736810:87 , srv648229297:87 , srv12615721:88 , srv1413034191:88 , srv1539683888:88 , srv1706228640:88 , srv1943236897:88 , srv301362638:88 , srv589780314:88 , srv630128150:88 , srv1662167050:89 , srv1120316696:530 }
2023-10-14 13:57:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-14 13:57:13,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv430089558=79, srv1706228640=45, srv326417666=76, srv432452400=80, srv12615721=10, srv133935008=18, srv2071209181=67, srv1507956776=28, srv876356254=98, srv1606396430=36, srv589780314=88, srv543844129=84, srv384678811=78, srv645317315=90, srv2045124652=66, srv168981059=43, srv582724493=87, srv830475214=94, srv1539683888=31, srv109943941=5, srv1270692772=11, srv1555255317=34, srv2044321696=65, srv1325259187=16, srv1028208265=3, srv1943236897=60, srv2027129669=63, srv1006584719=0, srv171349333=46, srv868331236=97, srv1821216564=53, srv1618314271=37, srv1644471815=40, srv1127776738=8, srv1129189678=9, srv149830880=26, srv1839816345=54, srv232983289=72, srv1816687253=52, srv445574129=82, srv2091283812=68, srv1804946621=51, srv1528550309=30, srv1477968373=25, srv1990602888=61, srv301362638=74, srv1749379941=49, srv1585016081=35, srv1650918008=41, srv727172727=92, srv1693265536=44, srv861238598=95, srv866015008=96, srv1452142038=23, srv1338509685=17, srv1727575309=47, srv36922699=77, srv1476096140=24, srv1397901949=20, srv1513128483=29, srv1318942747=15, srv877411892=99, srv162504894=38, srv127138438=12, srv1006590209=1, srv2039131977=64, srv545518205=85, srv21003249=69, srv200085113=62, srv263947563=73, srv486632795=83, srv1362069347=19, srv1628320217=39, srv1895483566=55, srv224156068=71, srv1913320832=57, srv437736810=81, srv1413034191=21, srv1303204096=13, srv1741322673=48, srv1942293077=59, srv737414076=93, srv1910726455=56, srv303662032=75, srv1120316696=7, srv1109624120=6, srv1923061313=58, srv1008554496=2, srv1311037666=14, srv2132601213=70, srv630128150=89, srv1503102391=27, srv579555538=86, srv648229297=91, srv1792292002=50, srv1098507372=4, srv154443427=32, srv1436699060=22, srv1662167050=42, srv1550443171=33} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-14 13:57:13,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-14 13:57:13,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-14 13:57:13,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-14 13:57:13,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-14 13:57:13,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-14 13:57:13,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-14 13:57:13,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-14 13:57:13,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-14 13:57:13,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-14 13:57:13,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-14 13:57:13,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-14 13:57:13,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-14 13:57:13,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-14 13:57:13,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-14 13:57:13,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-14 13:57:13,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-14 13:57:13,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-14 13:57:13,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-14 13:57:13,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-14 13:57:13,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-14 13:57:13,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-14 13:57:13,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-14 13:57:13,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-14 13:57:13,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-14 13:57:13,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-14 13:57:13,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-14 13:57:13,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-14 13:57:13,313 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-14 13:57:13,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-14 13:57:13,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-14 13:57:13,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-14 13:57:13,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-14 13:57:13,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-14 13:57:13,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-14 13:57:13,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-14 13:57:13,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-14 13:57:13,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-14 13:57:13,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-14 13:57:13,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-14 13:57:13,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-14 13:57:13,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-14 13:57:13,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-14 13:57:13,314 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-14 13:57:13,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-14 13:57:13,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-14 13:57:13,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-14 13:57:13,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-14 13:57:13,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-14 13:57:13,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-14 13:57:13,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-14 13:57:13,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-14 13:57:13,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-14 13:57:13,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-14 13:57:13,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-14 13:57:13,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-14 13:57:13,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-14 13:57:13,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-14 13:57:13,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-14 13:57:13,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-14 13:57:13,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-14 13:57:13,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-14 13:57:13,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-14 13:57:13,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-14 13:57:13,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-14 13:57:13,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-14 13:57:13,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-14 13:57:13,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-14 13:57:13,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-14 13:57:13,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-14 13:57:13,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-14 13:57:13,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-14 13:57:13,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-14 13:57:13,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-14 13:57:13,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-14 13:57:13,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-14 13:57:13,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-14 13:57:13,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-14 13:57:13,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-14 13:57:13,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-14 13:57:13,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-14 13:57:13,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-14 13:57:13,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-14 13:57:13,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-14 13:57:13,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-14 13:57:13,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-14 13:57:13,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-14 13:57:13,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-14 13:57:13,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-14 13:57:13,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-14 13:57:13,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-14 13:57:13,317 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-14 13:57:13,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-14 13:57:13,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-14 13:57:13,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-14 13:57:13,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-14 13:57:13,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-14 13:57:13,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-14 13:57:13,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-14 13:57:13,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-14 13:57:13,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-14 13:57:13,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-14 13:57:13,397 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00365626180754161); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006374049520922626); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00365626180754161); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-14 13:57:13,397 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-14 13:57:13,397 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.539888980391628E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00365626180754161); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006374049520922626); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00365626180754161); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-14 13:57:13,619 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 293 ms to try 5000 different iterations.  Found a solution that moves 238 regions; Going from a computed imbalance of 6.539888980391628E-4 to a new imbalance of 1.944106797983421E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.026444444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-14 13:57:13,620 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1413034191:83 , srv1452142038:83 , srv1585016081:83 , srv168981059:83 , srv1942293077:83 , srv2071209181:83 , srv301362638:83 , srv303662032:83 , srv1006590209:84 , srv1008554496:84 , srv1028208265:84 , srv1098507372:84 , srv1109624120:84 , srv12615721:84 , srv1270692772:84 , srv127138438:84 , srv1311037666:84 , srv1318942747:84 , srv1325259187:84 , srv133935008:84 , srv1397901949:84 , srv1436699060:84 , srv1477968373:84 , srv149830880:84 , srv1507956776:84 , srv1528550309:84 , srv1539683888:84 , srv1550443171:84 , srv1606396430:84 , srv1644471815:84 , srv1662167050:84 , srv1706228640:84 , srv1741322673:84 , srv1749379941:84 , srv1792292002:84 , srv1804946621:84 , srv1816687253:84 , srv1895483566:84 , srv1910726455:84 , srv1923061313:84 , srv1943236897:84 , srv1990602888:84 , srv200085113:84 , srv2044321696:84 , srv2045124652:84 , srv2091283812:84 , srv21003249:84 , srv2132601213:84 , srv263947563:84 , srv430089558:84 , srv437736810:84 , srv543844129:84 , srv579555538:84 , srv589780314:84 , srv645317315:84 , srv648229297:84 , srv727172727:84 , srv830475214:84 , srv861238598:84 , srv876356254:84 , srv109943941:85 , srv1127776738:85 , srv1129189678:85 , srv1303204096:85 , srv1338509685:85 , srv1362069347:85 , srv1476096140:85 , srv1503102391:85 , srv1513128483:85 , srv154443427:85 , srv1555255317:85 , srv1618314271:85 , srv162504894:85 , srv1628320217:85 , srv1650918008:85 , srv1693265536:85 , srv171349333:85 , srv1727575309:85 , srv1821216564:85 , srv1839816345:85 , srv1913320832:85 , srv2027129669:85 , srv2039131977:85 , srv224156068:85 , srv232983289:85 , srv326417666:85 , srv36922699:85 , srv384678811:85 , srv432452400:85 , srv445574129:85 , srv486632795:85 , srv545518205:85 , srv582724493:85 , srv630128150:85 , srv737414076:85 , srv866015008:85 , srv868331236:85 , srv877411892:85 , srv1006584719:86 , srv1120316696:652 }
2023-10-14 13:57:13,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-14 13:57:13,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv430089558=79, srv1706228640=45, srv326417666=76, srv432452400=80, srv12615721=10, srv133935008=18, srv2071209181=67, srv1507956776=28, srv876356254=98, srv1606396430=36, srv589780314=88, srv543844129=84, srv384678811=78, srv645317315=90, srv2045124652=66, srv168981059=43, srv582724493=87, srv830475214=94, srv1539683888=31, srv109943941=5, srv1270692772=11, srv1555255317=34, srv2044321696=65, srv1325259187=16, srv1028208265=3, srv1943236897=60, srv2027129669=63, srv1006584719=0, srv171349333=46, srv868331236=97, srv1821216564=53, srv1618314271=37, srv1644471815=40, srv1127776738=8, srv1129189678=9, srv149830880=26, srv1839816345=54, srv232983289=72, srv1816687253=52, srv445574129=82, srv2091283812=68, srv1804946621=51, srv1528550309=30, srv1477968373=25, srv1990602888=61, srv301362638=74, srv1749379941=49, srv1585016081=35, srv1650918008=41, srv727172727=92, srv1693265536=44, srv861238598=95, srv866015008=96, srv1452142038=23, srv1338509685=17, srv1727575309=47, srv36922699=77, srv1476096140=24, srv1397901949=20, srv1513128483=29, srv1318942747=15, srv877411892=99, srv162504894=38, srv127138438=12, srv1006590209=1, srv2039131977=64, srv545518205=85, srv21003249=69, srv200085113=62, srv263947563=73, srv486632795=83, srv1362069347=19, srv1628320217=39, srv1895483566=55, srv224156068=71, srv1913320832=57, srv437736810=81, srv1413034191=21, srv1303204096=13, srv1741322673=48, srv1942293077=59, srv737414076=93, srv1910726455=56, srv303662032=75, srv1120316696=7, srv1109624120=6, srv1923061313=58, srv1008554496=2, srv1311037666=14, srv2132601213=70, srv630128150=89, srv1503102391=27, srv579555538=86, srv648229297=91, srv1792292002=50, srv1098507372=4, srv154443427=32, srv1436699060=22, srv1662167050=42, srv1550443171=33} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-14 13:57:13,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-14 13:57:13,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-14 13:57:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-14 13:57:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-14 13:57:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-14 13:57:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-14 13:57:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-14 13:57:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-14 13:57:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-14 13:57:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-14 13:57:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-14 13:57:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-14 13:57:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-14 13:57:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-14 13:57:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-14 13:57:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-14 13:57:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-14 13:57:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-14 13:57:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-14 13:57:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-14 13:57:13,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-14 13:57:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-14 13:57:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-14 13:57:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-14 13:57:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-14 13:57:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-14 13:57:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-14 13:57:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-14 13:57:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-14 13:57:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-14 13:57:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-14 13:57:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-14 13:57:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-14 13:57:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-14 13:57:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-14 13:57:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-14 13:57:13,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-14 13:57:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-14 13:57:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-14 13:57:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-14 13:57:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-14 13:57:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-14 13:57:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-14 13:57:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-14 13:57:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-14 13:57:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-14 13:57:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-14 13:57:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-14 13:57:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-14 13:57:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-14 13:57:13,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-14 13:57:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-14 13:57:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-14 13:57:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-14 13:57:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-14 13:57:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-14 13:57:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-14 13:57:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-14 13:57:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-14 13:57:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-14 13:57:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-14 13:57:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-14 13:57:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-14 13:57:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-14 13:57:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-14 13:57:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-14 13:57:13,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-14 13:57:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-14 13:57:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-14 13:57:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-14 13:57:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-14 13:57:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-14 13:57:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-14 13:57:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-14 13:57:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-14 13:57:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-14 13:57:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-14 13:57:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-14 13:57:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-14 13:57:13,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-14 13:57:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-14 13:57:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-14 13:57:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-14 13:57:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-14 13:57:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-14 13:57:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-14 13:57:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-14 13:57:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-14 13:57:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-14 13:57:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-14 13:57:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-14 13:57:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-14 13:57:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-14 13:57:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-14 13:57:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-14 13:57:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-14 13:57:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-14 13:57:13,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-14 13:57:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-14 13:57:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-14 13:57:13,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-14 13:57:13,718 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.777455636076716E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-14 13:57:13,718 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1413034191:83 , srv1452142038:83 , srv1585016081:83 , srv168981059:83 , srv1942293077:83 , srv2071209181:83 , srv301362638:83 , srv303662032:83 , srv1006590209:84 , srv1008554496:84 , srv1028208265:84 , srv1098507372:84 , srv1109624120:84 , srv12615721:84 , srv1270692772:84 , srv127138438:84 , srv1311037666:84 , srv1318942747:84 , srv1325259187:84 , srv133935008:84 , srv1397901949:84 , srv1436699060:84 , srv1477968373:84 , srv149830880:84 , srv1507956776:84 , srv1528550309:84 , srv1539683888:84 , srv1550443171:84 , srv1606396430:84 , srv1644471815:84 , srv1662167050:84 , srv1706228640:84 , srv1741322673:84 , srv1749379941:84 , srv1792292002:84 , srv1804946621:84 , srv1816687253:84 , srv1895483566:84 , srv1910726455:84 , srv1923061313:84 , srv1943236897:84 , srv1990602888:84 , srv200085113:84 , srv2044321696:84 , srv2045124652:84 , srv2091283812:84 , srv21003249:84 , srv2132601213:84 , srv263947563:84 , srv430089558:84 , srv437736810:84 , srv543844129:84 , srv579555538:84 , srv589780314:84 , srv645317315:84 , srv648229297:84 , srv727172727:84 , srv830475214:84 , srv861238598:84 , srv876356254:84 , srv109943941:85 , srv1127776738:85 , srv1129189678:85 , srv1303204096:85 , srv1338509685:85 , srv1362069347:85 , srv1476096140:85 , srv1503102391:85 , srv1513128483:85 , srv154443427:85 , srv1555255317:85 , srv1618314271:85 , srv162504894:85 , srv1628320217:85 , srv1650918008:85 , srv1693265536:85 , srv171349333:85 , srv1727575309:85 , srv1821216564:85 , srv1839816345:85 , srv1913320832:85 , srv2027129669:85 , srv2039131977:85 , srv224156068:85 , srv232983289:85 , srv326417666:85 , srv36922699:85 , srv384678811:85 , srv432452400:85 , srv445574129:85 , srv486632795:85 , srv545518205:85 , srv582724493:85 , srv630128150:85 , srv737414076:85 , srv866015008:85 , srv868331236:85 , srv877411892:85 , srv1006584719:86 , srv1120316696:652 }
2023-10-14 13:57:13,745 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=268 (was 266) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=3734 (was 4218)
2023-10-14 13:57:13,751 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=268, ProcessCount=168, AvailableMemoryMB=3733
2023-10-14 13:57:13,752 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-10-14 13:57:13,752 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-10-14 13:57:13,763 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1037569441:84 , srv1115558038:84 , srv1141397096:84 , srv1142551166:84 , srv1146684442:84 , srv1255942472:84 , srv1259937542:84 , srv1274283632:84 , srv1348067729:84 , srv1375237802:84 , srv1440459295:84 , srv1503858725:84 , srv1566111080:84 , srv1642174384:84 , srv1676351707:84 , srv2015463735:84 , srv2085373459:84 , srv2114955151:84 , srv261342726:84 , srv39192224:84 , srv467560421:84 , srv611910979:84 , srv61484171:84 , srv665065406:84 , srv679636095:84 , srv737027927:84 , srv762405148:84 , srv969749116:84 , srv974262717:84 , srv2025181930:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-10-14 13:57:13,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-14 13:57:13,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1566111080=12, srv1375237802=9, srv679636095=25, srv1440459295=10, srv1274283632=7, srv1676351707=14, srv2114955151=18, srv2025181930=16, srv1503858725=11, srv665065406=24, srv762405148=27, srv467560421=21, srv1348067729=8, srv1142551166=3, srv611910979=22, srv737027927=26, srv261342726=19, srv1642174384=13, srv1141397096=2, srv974262717=29, srv1259937542=6, srv1037569441=0, srv969749116=28, srv1146684442=4, srv39192224=20, srv2085373459=17, srv1115558038=1, srv61484171=23, srv2015463735=15, srv1255942472=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-14 13:57:13,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-14 13:57:13,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-14 13:57:13,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-14 13:57:13,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-14 13:57:13,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-14 13:57:13,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-14 13:57:13,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-14 13:57:13,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-14 13:57:13,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-14 13:57:13,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-14 13:57:13,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-14 13:57:13,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-14 13:57:13,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-14 13:57:13,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-14 13:57:13,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-14 13:57:13,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-14 13:57:13,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-14 13:57:13,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-14 13:57:13,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-14 13:57:13,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-14 13:57:13,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-14 13:57:13,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-14 13:57:13,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-14 13:57:13,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-14 13:57:13,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-14 13:57:13,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-14 13:57:13,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-14 13:57:13,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-14 13:57:13,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-14 13:57:13,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-14 13:57:13,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-10-14 13:57:13,802 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0821174657934904, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-14 13:57:13,802 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912451792717614, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0821174657934904, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-10-14 13:59:13,784 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9548362 different iterations.  Found a solution that moves 1834 regions; Going from a computed imbalance of 0.9912451792717614 to a new imbalance of 4.280647492113412E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6792592592592592, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-14 13:59:13,790 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1037569441:90 , srv1115558038:90 , srv1141397096:90 , srv1142551166:90 , srv1146684442:90 , srv1255942472:90 , srv1259937542:90 , srv1274283632:90 , srv1348067729:90 , srv1375237802:90 , srv1440459295:90 , srv1503858725:90 , srv1566111080:90 , srv1642174384:90 , srv1676351707:90 , srv2015463735:90 , srv2025181930:90 , srv2085373459:90 , srv2114955151:90 , srv261342726:90 , srv39192224:90 , srv467560421:90 , srv611910979:90 , srv61484171:90 , srv665065406:90 , srv679636095:90 , srv737027927:90 , srv762405148:90 , srv969749116:90 , srv974262717:90 }
2023-10-14 13:59:13,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-14 13:59:13,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1566111080=12, srv1375237802=9, srv679636095=25, srv1440459295=10, srv1274283632=7, srv1676351707=14, srv2114955151=18, srv2025181930=16, srv1503858725=11, srv665065406=24, srv762405148=27, srv467560421=21, srv1348067729=8, srv1142551166=3, srv611910979=22, srv737027927=26, srv261342726=19, srv1642174384=13, srv1141397096=2, srv974262717=29, srv1259937542=6, srv1037569441=0, srv969749116=28, srv1146684442=4, srv39192224=20, srv2085373459=17, srv1115558038=1, srv61484171=23, srv2015463735=15, srv1255942472=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-14 13:59:13,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-14 13:59:13,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-14 13:59:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-14 13:59:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-14 13:59:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-14 13:59:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-14 13:59:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-14 13:59:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-14 13:59:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-14 13:59:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-14 13:59:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-14 13:59:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-14 13:59:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-14 13:59:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-14 13:59:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-14 13:59:13,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-14 13:59:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-14 13:59:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-14 13:59:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-14 13:59:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-14 13:59:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-14 13:59:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-14 13:59:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-14 13:59:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-14 13:59:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-14 13:59:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-14 13:59:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-14 13:59:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-14 13:59:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-14 13:59:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-14 13:59:13,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-10-14 13:59:13,807 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-14 13:59:13,807 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1037569441:90 , srv1115558038:90 , srv1141397096:90 , srv1142551166:90 , srv1146684442:90 , srv1255942472:90 , srv1259937542:90 , srv1274283632:90 , srv1348067729:90 , srv1375237802:90 , srv1440459295:90 , srv1503858725:90 , srv1566111080:90 , srv1642174384:90 , srv1676351707:90 , srv2015463735:90 , srv2025181930:90 , srv2085373459:90 , srv2114955151:90 , srv261342726:90 , srv39192224:90 , srv467560421:90 , srv611910979:90 , srv61484171:90 , srv665065406:90 , srv679636095:90 , srv737027927:90 , srv762405148:90 , srv969749116:90 , srv974262717:90 }
2023-10-14 13:59:13,818 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=175 (was 268), ProcessCount=165 (was 168), AvailableMemoryMB=3376 (was 3733)